For our fourth live chat we had not just one, but three extremely kind folks who joined us online to talk about how best to design a scientific experiment. Patrick, Suzi and Rob all and in different ways study the patterns, causes and effects of disease. They design really big studies and make sure they get reliable results out of them, and that they answer the right questions.
Below is a summary of some of the most interesting points that were made and discussed during the chat.
Q: How do we avoid bias or the appearance of bias, as we’re all cloth using/selling/advising type people?
Patrick: This is a great question and one that is very relevant for the area that I work in – clinical trials of new treatments. There is always a new treatment that people think is going to be great, and so a clinical trial has to be conducted in such a way that the results are adequate to convince skeptics. One way is to use objective outcomes, that is measurements that are likely to be free from bias, such as a laboratory measurement, as opposed to subjective outcomes such as pain or whether a nappy is “clean”.
Suzi: Also double blind studies are a good way of reducing bias. A double blind study is when both the people in the experiment AND the people analysing the results do not know which condition they were in (i.e. with different ways of washing them).
Rob: But since there isn’t always a way to avoid bias, it’s also important to be clear with yourself that there is that potential and to acknowledge it.
Q: We can get some lab testing done to get some objective measures of how clean a nappy is – like how much bacteria is present in the nappy. But that is expensive and there’s a limit to how many nappies we could test. We’ve realised there are loads of variables (hard/soft water, type of washing machine, types of nappies) – how do we work out how many nappies we need to test to get a reliable result?
Patrick: This is a statistical question. To get to the answer, you really need to work backwards from what is the scientific question being asked – and what is the outcome that is going to determine whether the study worked. Another way of looking at it is: What sort of difference would actually make it worth it to use the more expensive/less environmentally friendly option. This is called the minimum clinically important difference (for a medical study). ie the minimum difference that is actually going to be of interest.
Suzi: When you’re working out how many participants you need in a study, you do what’s called a power calculation. But what you really need is some idea of the size of the difference you’d expect. Like Patrick said, it’s about working out how many people/nappies we’ll need in each condition to conduct a statistically meaningful analysis. Knowledge of previous studies that have looked at similar things would be useful.
Q: I’ve always wondered this, especially buying second hand nappies: are the bacteria left on the nappy going to do any harm? I know people have had a hard time clearing things like thrush from cloth nappies and often say as soon as the baby goes back into the nappy this comes back. So we were thinking about sending nappies to a lab to check bacterial growth, do you think this would be the right approach?
Patrick: In a trial, we always try to choose an outcome that is relevant to the patient. ie something that affects how the patient feels, functions or survives, rather than just a lab measure. So I would suggest that you go with an outcome that is relevant to the parents/kids such as thrush rather than number of bacteria. Going back to the question of avoiding bias – if you could get an objective definition of thrush (for example as a rash of x size), that would a be a good measure for a study.
Penny Broderic (from Malvern Nappy Advisory Service) pointed out that a definitive answer on “Is 60 degree cotton wash enough to significantly reduce the chance of thrush issues passing to the next customer/client’ s child?” would be extremely useful to the libraries.
Q: For me the outcomes that are most relevant to the user are: absorbency, smell and hygiene. How could I quantify these?
Patrick: I’m not sure how you would quantify smell objectively, but for absorbency I guess you need a fabric expert on how to measure that. Hygiene – it comes back to the question you asked earlier, you could go with a lab measurement or a more participant-relevant outcome such as incidence of rash.
Q: How do we deal with people’s different perception of what “clean” means for everyone? Would we need a relatively large sample size (e.g. we’ve got 300 and something people in the group) to even out differences in people’s perceptions for that?
Patrick: 300 participants is a good number. You could do a study where each participant tries a different method for a week.
At this point the conversation was moved to ALSPAC, which is what Suzi works with. ALSPAC is a massive study where scientists have collected info about over 14,000 pregnant women and the babies they had in 1991 and 1992. They’ve been following these children ever since and now even their children’s children.
Q: How much information can ALSPAC give us on the effect of using reusable nappies on hip development and walking?
Suzi: The important thing to think about when using data like this, is that you’re not randomly assigning people to different groups. This means people will have chosen to use the type of nappy they use. And people who choose to use cloth nappies might be different from people who don’t, in lots of other ways.
Penny added that we have to keep in mind that nappies from 4 years ago are different from the modern ones.
Q: Could ALSPAC be used to find if there’s a link between cloth nappy use and earlier potty training?
Suzi: I reckon that’s possible to look at too. There’s a great deal of data that was collected about pregnancy and early infancy. That was what the cohort was initially set up for.
Patrick: Parents who use cloth nappies might tend to be more involved parents and therefore be keen to try and persevere with potty training earlier. Might not be a causal link with cloth nappies. This is obviously the point Suzi made about the limitations of observational data.
And then, to finish off the chat, we asked the experts whether they had any words of wisdom they’d like to leave us with. And what would they say is key to bear in mind when designing an experiment?
Patrick: I would say randomisation is really important, as is writing a protocol up front, articulating the methods for the experiment clearly and documenting beforehand.
Suzi: I think you’re all thinking about the issues that surround the questions that you’re interested in! But don’t get so bogged down in limitations that you get disheartened!
Rob: Yes, I agree. And it’s also really important to choose an outcome that’s relevant to the study participants and, wherever possible, objective.
This week’s online chat with Patrick, Suzi and Rob was quite intense and full of useful tips which will hopefully help us in the future design of our own experiments. Thanks so much to them for donating their evening to our cause, we really appreciate it!