The $28 Billion a Year Research Reproducibility Crisis

Comments Off on The $28 Billion a Year Research Reproducibility Crisis
The $28 Billion a Year Research Reproducibility Crisis

One of the foundations of the scientific method is that the results obtained from experiments are reproducible—that is, another team of researchers, using the same methods, should be able to obtain the same results. If the results are not reproducible, the findings are not valid.

Tragically, the lack of reproducible results in medical research is not the rare occurrence that most non-scientists assume it to be. In fact, the problem is so widespread that billions of dollars are wasted each year, while millions of patients are being deprived of the breakthroughs that could be discovered if the studies were more rigorous.

Earlier in this issue, in the trend Moving from “Good Intentions” to “Great Results,” we mentioned that scientists from the biotech firm Amgen were not able to reproduce the results reported in 47 of the 53 most influential research papers that have inspired billions of dollars in investments in cancer research.

Other investigators have made similarly alarming discoveries. In the journal Nature Reviews Drug Discovery, researchers reported that only 6 percent of medical experiments were completely reproducible. And in recent years, Phase II studies of potential new drugs, which are designed to test the results of Phase I studies, have succeeded in reproducing those results at a rate of only 18 percent.1

Now, a new study on the causes and costs of irreproducible research, published in the journal PLoS Biology, estimates conservatively that American scientists spend $28 billion each year on basic biomedical research that cannot be repeated successfully.2

Leonard Freedman, head of the non-profit Global Biological Standards Institute, along with Boston University economists Iain Cockburn and Timothy Simcoe, explained the most common mistakes that lead to irreproducible results. They defined irreproducibility as occurring when any mistakes or omissions keep other researchers from replicating the original findings.

Freedman and his colleagues looked at four common sources of irreproducibility:

  • Study design
  • Laboratory protocols
  • Biological reagents and reference materials
  • Data analysis and reporting

They concluded that the biggest problem was the use of poor materials, at 36 percent, followed by poor study design at 28 percent, and flawed data analysis at 26 percent...

To continue reading, become a paid subscriber for full access.
Already a Trends Magazine subscriber? Login for full access now.

Subscribe for as low as $195/year

  • Get 12 months of Trends that will impact your business and your life
  • Gain access to the entire Trends Research Library
  • Optional Trends monthly CDs in addition to your On-Line access
  • Receive our exclusive "Trends Investor Forecast 2015" as a free online gift
  • If you do not like what you see, you can cancel anytime and receive a 100% full refund