Can up to 70% of scientific studies not be reproduced?

Question

I read and heard a lot about the Reproducibility Initiative recently, claiming that the data of many scientific studies cannot/was not/is not be reproduced.

“In the last year, problems in reproducing academic research have drawn a lot of public attention, particularly in the context of translating research into medical advances. Recent studies indicate that up to 70% of research from academic labs cannot be reproduced, representing an enormous waste of money and effort,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “In my experience as a researcher, I found that the problem lay primarily in the lack of incentives and opportunities for validation—the Reproducibility Initiative directly tackles these missing pieces.”

Unfortunately I was not able to find those studies (where these reproduced!?) proving this statement. I want to know where these studies where carried out, medicine, biology, psychology, but couldn't find anything. I'm also somehow skeptical that science is in that bad shape, considering that studies are often used/mandatory here on skeptics.se for good answers and to get license for pharmaceutical products. 70% looks a bit too high to me.

Answer 1

ALS Therapy Development Institute re-tested 70+ drugs from 221 independent studies:

• 0 reproduced (1)
• Minocycline: effective in four separate ALS mouse studies worsened symptoms in a clinical trial of more than 400 patients (2)

National Institute of Neurological Disorders and Stroke (NINDS) conducted sponsored replication of 12 spinal cord injury studies:

• 2/12 successfully reproduced (3)

Bayer conducted in-house target validation studies

• 14/67 reproduced (4)

Amgen attempted to reproduce 53 “landmark” oncology publications:

• 6/53 reproduced (5)

References

1. Scott et al. Amyotroph Lateral Scler. 9, 4-15 (2008).
2. Gordon et al. Lancet Neurol. 6, 1045–1053 (2007).
3. Stuart et al. Experimental Neurology 233, 597–605 (2012).
4. Prinz et al. Nat Rev Drug Discov. 10, 712 (2011).
5. Begley and Ellis. Nature. 483, 531-3 (2012).

Answer 2

There are several studies that document the problems of reproducibility, and thoroughly review the issue in indications of medicine, oncology, and neuroscience.

The two separate publications below detail how over 2/3 of landmark oncology studies were found not to be reproducible.

Asadullah: http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html

Glenn Begley / Lee Ellis: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html

Answer 3

Yes, and the problem exists across many areas of science

The problem of reproducibility is attracting a lot more attention. A recent Nature News Feature discusses the possible causes of bad results in the literature and many examples where they have been exposed. Its headline sums up the problem:

Humans are remarkably good at self-deception. But growing concern about reproducibility is driving many researchers to seek ways to fight their own worst instincts.

One huge problem is that results that sound plausible get a lot less scrutiny than unusual results so wrong but plausible results get a free pass on error checking far too often.

The article summarises a range of different reproducibility experiments and the results suggest that the majority of results across many fields are not reproducible (it is worth reading the article for all the links to these studies).

Failure to understand our own biases has helped to create a crisis of confidence about the reproducibility of published results, says statistician John Ioannidis, co-director of the Meta-Research Innovation Center at Stanford University in Palo Alto, California. The issue goes well beyond cases of fraud. Earlier this year, a large project that attempted to replicate 100 psychology studies managed to reproduce only slightly more than one-third. In 2012, researchers at biotechnology firm Amgen in Thousand Oaks, California, reported that they could replicate only 6 out of 53 landmark studies in oncology and haematology. And in 2009, Ioannidis and his colleagues described how they had been able to fully reproduce only 2 out of 18 microarray-based gene-expression studies.

On a more positive note the article recommends a number of ways to de-bias scientific answers. Some of these are summarised in the picture below:

So there are plenty of known reasons why bad results get published and there are a lot of them. But it is something that good scientists can fight.