So I’ve been away for what appears to be the last week of summer here. Sigh. But I am back with a previously prepared article! I recently entered a writing competition in Nature. Unfortunately I didn’t win a place at the meeting or to have my writing feature on the blog, so it means that I can publish it here. There were a number of topics to write about and I chose to discuss reproducibility in science.
What are the benefits of reproducibility in science? That there needs to be the question of what the benefit of reproducibility is, indicates that there are currently not only problems in reproducibility, but in attitudes towards producing reproducible data.
What is reproducibility?
What do we mean by ‘reproducibility’? It should mean an ability to repeat an experiment and obtain the same result, however it also means that the same conclusion is reached in different experiments by different people.
One should ask how does an experiment or a series of experiments fail to be reproducible? In this sense it can be broken down into the different factors that influence the scientific process: 1) experimental design; 2) techniques and methods; and 3) fitting experiments to a ‘story’ for publication rather than presenting and discussing data ‘as is’.
Experimental design consists of the question or hypothesis that is being asked, and how experiments are planned around this question. Considerations for experimental design should include whether the experiments will be able to be repeated to confirm the results. Often, studies are performed that cannot be repeated due to cost, time and logistics. Furthermore, factors that are often overlooked include sample size, randomization and the effect of sex-differences.
Problems with experimental techniques and methods can arise when experimental procedures are not properly recorded, particularly small deviations from a protocol. Issues in reproducibility can also occur when people in the same laboratory each have their own protocol for a technique and thus there is neither cohesion nor an agreed lab protocol. These differences can obviously arise due to the constant turn over of students and post docs. And, what about a technique that is so laborious, flawed and unreliable that only one person in the lab is able to get it to work?!
Finally, data and results will often be used to ‘fit’ a story. This means that in some cases, data will not be included if it does not fit to the story or idea being pursued. Frequently, these results can be contradictory to the data presented, are ‘negative’ or ‘neutral’ results; or if included would indicate a flawed scientific method. All too often, data is used to fit to a story based on ‘novelty’ rather than a small, but important contribution to a field. Included in fitting data to a story is ‘P-hacking’, a term coined to explain the phenomenon of performing different statistical tests until a significant P-value is obtained, or removing multiple replicates to achieve a significant result. What can happen in this case, is that inherent biological variability is overlooked or not reported, meaning that the results become less reproducible due to the variability.
So if we consider how science can be irreproducible, then a benefit of reproducibility will be an improved scientific method. Encouraging researchers to consider reproducibility before they begin an experiment will result in meticulous, well planned and executed experiments and lead to a reduction in the wastage of time and money.
The other benefit will be how science, and the use of the money given to science, is viewed by the general public. Improving the scientific method to place a high emphasis on reproducibility (which should always be the consideration) rather than on the ‘novelty’ of the result or the impact factor of the journal the data is published in, will gain public trust not only of the research but also of the researchers.
The final thought is that better reproducibility will improve publishing and communication between scientists. After all, each manuscript and experiment should contribute to a field and our knowledge, not knock another scientist off the shelf.
No sources this week as this was an opinion piece, however the link below is to an interesting article.
Begley, C. Glenn, and John PA Ioannidis. “Reproducibility in science improving the standard for basic and preclinical research.” Circulation research 116.1 (2015): 116-126.