Take the 2-minute tour ×
Skeptics Stack Exchange is a question and answer site for scientific skepticism. It's 100% free, no registration required.

I read and heard a lot about the Reproducibility Initiative recently, claiming that the data of many scientific studies cannot/was not/is not be reproduced.

“In the last year, problems in reproducing academic research have drawn a lot of public attention, particularly in the context of translating research into medical advances. Recent studies indicate that up to 70% of research from academic labs cannot be reproduced, representing an enormous waste of money and effort,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “In my experience as a researcher, I found that the problem lay primarily in the lack of incentives and opportunities for validation—the Reproducibility Initiative directly tackles these missing pieces.”

Unfortunately I was not able to find those studies (where these reproduced!?) proving this statement. I want to know where these studies where carried out, medicine, biology, psychology, but couldn't find anything. I'm also somehow skeptical that science is in that bad shape, considering that studies are often used/mandatory here on skeptics.se for good answers and to get license for pharmaceutical products. 70% looks a bit too high to me.

share|improve this question
5  
I really don't find this all that shocking, no ones pays money for people to go find out things that are already known. There is no fame in being the fact checker, nobody gets a Nobel Prize for "best recreation of previous research." –  Ryathal Nov 5 '12 at 20:59
2  
Ryathal, while I am sympathetic to your view it is not completely true. When the Italians claimed that neutrinos traveled faster than light a flurry of experiments were done to reproduce the results. See here for the details: news.sciencemag.org/scienceinsider/2012/02/… –  thisfeller Nov 5 '12 at 21:24
2  
You normally become the fact checker if you have solid claims that the results of a given study are either erroneous or incomplete. It's no wonder everyone tried to reproduce the italians' experiment, since faster than light speeds are thought to be impossible (further tests revealed that there are no significant differences between the speed of light and that of neutrinos either). Some discoveries are just so potentially groundbreaking that they are immediately going to be verified. Others, not as flamboyant, can be somewhat expected by scientists and will often be taken at face value. –  Dungarth Nov 5 '12 at 21:50
1  
@thisfeller The OPERA guys never believed that number and they phrased the paper very carefully (it's worth reading to see exactly what the claim). They'd been sitting on it while they tried to figure it out, but it leaked and then they had to say something. In any case, in particle physics we tend to reproduce the results of the nth generation machine as part of commissioning the n+1st generation, so there is a strong expectation that your work will be put to the test. On the other hand we have some anomalous results in our history, too. –  dmckee Nov 5 '12 at 23:39
1  
We do heavy-ion research, Tier-2 for the latest from the LHC. –  thisfeller Nov 6 '12 at 2:34
show 3 more comments

3 Answers

up vote 36 down vote accepted

ALS Therapy Development Institute re-tested 70+ drugs from 221 independent studies:

  • 0 reproduced (1)
  • Minocycline: effective in four separate ALS mouse studies worsened symptoms in a clinical trial of more than 400 patients (2)

National Institute of Neurological Disorders and Stroke (NINDS) conducted sponsored replication of 12 spinal cord injury studies:

  • 2/12 successfully reproduced (3)

Bayer conducted in-house target validation studies

  • 14/67 reproduced (4)

Amgen attempted to reproduce 53 “landmark” oncology publications:

  • 6/53 reproduced (5)

References

  1. Scott et al. Amyotroph Lateral Scler. 9, 4-15 (2008).
  2. Gordon et al. Lancet Neurol. 6, 1045–1053 (2007).
  3. Stuart et al. Experimental Neurology 233, 597–605 (2012).
  4. Prinz et al. Nat Rev Drug Discov. 10, 712 (2011).
  5. Begley and Ellis. Nature. 483, 531-3 (2012).
share|improve this answer
    
Thanks Dr. Iorns for the quick and detailed reply, that are the references I was missing on many blogs and news sites explaining your initiative recently. I would put those references on your website, so it becomes clear it's esp. a major problem in medical sciences. –  Hauser Nov 5 '12 at 22:04
    
This would also make a good answer to skeptics.stackexchange.com/questions/1973/… –  matt_black Nov 6 '12 at 8:28
    
Also raises questions about evidence-based medicine, if the evidence has been cooked: skeptics.stackexchange.com/questions/3111/… –  mmr Nov 6 '12 at 22:07
add comment

There are several studies that document the problems of reproducibility, and thoroughly review the issue in indications of medicine, oncology, and neuroscience.

The two separate publications below detail how over 2/3 of landmark oncology studies were found not to be reproducible.

Asadullah: http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html

Glenn Begley / Lee Ellis: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html

share|improve this answer
    
Thanks to you too, someone informed both of you really quick ;) –  Hauser Nov 5 '12 at 22:05
1  
To make this a better answer, could you summarise the content of the linked papers? –  matt_black Nov 8 '12 at 23:05
add comment

This is indeed a major problem in many academic fields, mine included (just finishing my PhD in Aerospace Engineering), as reproducibility is one of the basic pillar of the scientific method. There are several issues that could thought as the main causes:

  • in universities, most of the research is done by temporary people, i.e. students or post-doc. While younger professors are still very much involved in the research, some tenured professors have so many other engagements and have built such a large group that they cannot possibly keep up with everything going on in their lab. Which makes transmission of knowledge problematic, especially because...
  • it's getting tougher and tougher to get to the new "stuff" that one requires to publish/get a doctorate. What I mean is that you don't know that much more today at 18 that someone knew at 18 25 years ago. Which means the path from what you know when you arrive in grad school to the stuff that is new is getting longer and more complicated by the minute. Pr Van Leer had a great analogy about that and I tried to poorly reproduce it here:

Path to PhD

This is the price we have to pay as we keep expanding the body of knowledge, however, I'm not sure our academic system is particularly well equipped to deal with this issue. This article details a few issues with that expanding knowledge and specifically how the sheer volume of publications means most of it cannot be materially reproduced, there is just no available time/money/people/resources to do it. The rate of publications would have to slow down for this to happen and then more recognition should be given to fact checkers like the Reproducibility Initiative.

As a consequence, it is extremely difficult to find the time and money to store/archive/document all the data produced so that your work is reproducible 1, 2, 5 years from now. These ArsTechnica article have a few references about how difficult it is to store data long-term: for example, the rate at which DNA sequencing produces data is larger than the rate of growth of hard drive space.

These problems are probably more serious in universities and less present in national labs with permanent staff and more stable funding. But if most of the studies at stake here come from universities, I'm not terribly surprised, albeit saddened, by that 70% number.

share|improve this answer
1  
Welcome to Skeptics! Please provide some references to support your claims. –  Sklivvz Nov 6 '12 at 12:25
    
Thanks for the advice, I'm gonna work on it. –  FrenchKheldar Nov 6 '12 at 15:18
    
While I realize that there are no sources for this, this is a very concise summary of modern academics, at least in the US. I can only speak anecdotally, but this post matches my experiences exactly. –  mmr Nov 7 '12 at 3:42
add comment

This post does not cite any references or sources. Please help improve this post by adding citations to reliable sources. Unsourced material may be challenged and removed.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.