Take the 2-minute tour ×
Skeptics Stack Exchange is a question and answer site for scientific skepticism. It's 100% free, no registration required.

I read and heard a lot about the Reproducibility Initiative recently, claiming that the data of many scientific studies cannot/was not/is not be reproduced.

“In the last year, problems in reproducing academic research have drawn a lot of public attention, particularly in the context of translating research into medical advances. Recent studies indicate that up to 70% of research from academic labs cannot be reproduced, representing an enormous waste of money and effort,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “In my experience as a researcher, I found that the problem lay primarily in the lack of incentives and opportunities for validation—the Reproducibility Initiative directly tackles these missing pieces.”

Unfortunately I was not able to find those studies (where these reproduced!?) proving this statement. I want to know where these studies where carried out, medicine, biology, psychology, but couldn't find anything. I'm also somehow skeptical that science is in that bad shape, considering that studies are often used/mandatory here on skeptics.se for good answers and to get license for pharmaceutical products. 70% looks a bit too high to me.

share|improve this question
6  
I really don't find this all that shocking, no ones pays money for people to go find out things that are already known. There is no fame in being the fact checker, nobody gets a Nobel Prize for "best recreation of previous research." –  Ryathal Nov 5 '12 at 20:59
2  
Ryathal, while I am sympathetic to your view it is not completely true. When the Italians claimed that neutrinos traveled faster than light a flurry of experiments were done to reproduce the results. See here for the details: news.sciencemag.org/scienceinsider/2012/02/… –  thisfeller Nov 5 '12 at 21:24
3  
You normally become the fact checker if you have solid claims that the results of a given study are either erroneous or incomplete. It's no wonder everyone tried to reproduce the italians' experiment, since faster than light speeds are thought to be impossible (further tests revealed that there are no significant differences between the speed of light and that of neutrinos either). Some discoveries are just so potentially groundbreaking that they are immediately going to be verified. Others, not as flamboyant, can be somewhat expected by scientists and will often be taken at face value. –  Dungarth Nov 5 '12 at 21:50
1  
@thisfeller The OPERA guys never believed that number and they phrased the paper very carefully (it's worth reading to see exactly what the claim). They'd been sitting on it while they tried to figure it out, but it leaked and then they had to say something. In any case, in particle physics we tend to reproduce the results of the nth generation machine as part of commissioning the n+1st generation, so there is a strong expectation that your work will be put to the test. On the other hand we have some anomalous results in our history, too. –  dmckee Nov 5 '12 at 23:39
1  
We do heavy-ion research, Tier-2 for the latest from the LHC. –  thisfeller Nov 6 '12 at 2:34

2 Answers 2

up vote 46 down vote accepted

ALS Therapy Development Institute re-tested 70+ drugs from 221 independent studies:

  • 0 reproduced (1)
  • Minocycline: effective in four separate ALS mouse studies worsened symptoms in a clinical trial of more than 400 patients (2)

National Institute of Neurological Disorders and Stroke (NINDS) conducted sponsored replication of 12 spinal cord injury studies:

  • 2/12 successfully reproduced (3)

Bayer conducted in-house target validation studies

  • 14/67 reproduced (4)

Amgen attempted to reproduce 53 “landmark” oncology publications:

  • 6/53 reproduced (5)

References

  1. Scott et al. Amyotroph Lateral Scler. 9, 4-15 (2008).
  2. Gordon et al. Lancet Neurol. 6, 1045–1053 (2007).
  3. Stuart et al. Experimental Neurology 233, 597–605 (2012).
  4. Prinz et al. Nat Rev Drug Discov. 10, 712 (2011).
  5. Begley and Ellis. Nature. 483, 531-3 (2012).
share|improve this answer
    
Thanks Dr. Iorns for the quick and detailed reply, that are the references I was missing on many blogs and news sites explaining your initiative recently. I would put those references on your website, so it becomes clear it's esp. a major problem in medical sciences. –  Hauser Nov 5 '12 at 22:04
    
This would also make a good answer to skeptics.stackexchange.com/questions/1973/… –  matt_black Nov 6 '12 at 8:28
2  
Also raises questions about evidence-based medicine, if the evidence has been cooked: skeptics.stackexchange.com/questions/3111/… –  mmr Nov 6 '12 at 22:07

There are several studies that document the problems of reproducibility, and thoroughly review the issue in indications of medicine, oncology, and neuroscience.

The two separate publications below detail how over 2/3 of landmark oncology studies were found not to be reproducible.

Asadullah: http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html

Glenn Begley / Lee Ellis: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html

share|improve this answer
    
Thanks to you too, someone informed both of you really quick ;) –  Hauser Nov 5 '12 at 22:05
2  
To make this a better answer, could you summarise the content of the linked papers? –  matt_black Nov 8 '12 at 23:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.