Skeptics Stack Exchange is a question and answer site for scientific skepticism. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I read and heard a lot about the Reproducibility Initiative recently, claiming that the data of many scientific studies cannot/was not/is not be reproduced.

“In the last year, problems in reproducing academic research have drawn a lot of public attention, particularly in the context of translating research into medical advances. Recent studies indicate that up to 70% of research from academic labs cannot be reproduced, representing an enormous waste of money and effort,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “In my experience as a researcher, I found that the problem lay primarily in the lack of incentives and opportunities for validation—the Reproducibility Initiative directly tackles these missing pieces.”

Unfortunately I was not able to find those studies (where these reproduced!?) proving this statement. I want to know where these studies where carried out, medicine, biology, psychology, but couldn't find anything. I'm also somehow skeptical that science is in that bad shape, considering that studies are often used/mandatory here on skeptics.se for good answers and to get license for pharmaceutical products. 70% looks a bit too high to me.

share|improve this question
7  
I really don't find this all that shocking, no ones pays money for people to go find out things that are already known. There is no fame in being the fact checker, nobody gets a Nobel Prize for "best recreation of previous research." – Ryathal Nov 5 '12 at 20:59
2  
Ryathal, while I am sympathetic to your view it is not completely true. When the Italians claimed that neutrinos traveled faster than light a flurry of experiments were done to reproduce the results. See here for the details: news.sciencemag.org/scienceinsider/2012/02/… – thisfeller Nov 5 '12 at 21:24
3  
You normally become the fact checker if you have solid claims that the results of a given study are either erroneous or incomplete. It's no wonder everyone tried to reproduce the italians' experiment, since faster than light speeds are thought to be impossible (further tests revealed that there are no significant differences between the speed of light and that of neutrinos either). Some discoveries are just so potentially groundbreaking that they are immediately going to be verified. Others, not as flamboyant, can be somewhat expected by scientists and will often be taken at face value. – Dungarth Nov 5 '12 at 21:50
1  
@thisfeller The OPERA guys never believed that number and they phrased the paper very carefully (it's worth reading to see exactly what the claim). They'd been sitting on it while they tried to figure it out, but it leaked and then they had to say something. In any case, in particle physics we tend to reproduce the results of the nth generation machine as part of commissioning the n+1st generation, so there is a strong expectation that your work will be put to the test. On the other hand we have some anomalous results in our history, too. – dmckee Nov 5 '12 at 23:39
1  
We do heavy-ion research, Tier-2 for the latest from the LHC. – thisfeller Nov 6 '12 at 2:34
up vote 47 down vote accepted

ALS Therapy Development Institute re-tested 70+ drugs from 221 independent studies:

  • 0 reproduced (1)
  • Minocycline: effective in four separate ALS mouse studies worsened symptoms in a clinical trial of more than 400 patients (2)

National Institute of Neurological Disorders and Stroke (NINDS) conducted sponsored replication of 12 spinal cord injury studies:

  • 2/12 successfully reproduced (3)

Bayer conducted in-house target validation studies

  • 14/67 reproduced (4)

Amgen attempted to reproduce 53 “landmark” oncology publications:

  • 6/53 reproduced (5)

References

  1. Scott et al. Amyotroph Lateral Scler. 9, 4-15 (2008).
  2. Gordon et al. Lancet Neurol. 6, 1045–1053 (2007).
  3. Stuart et al. Experimental Neurology 233, 597–605 (2012).
  4. Prinz et al. Nat Rev Drug Discov. 10, 712 (2011).
  5. Begley and Ellis. Nature. 483, 531-3 (2012).
share|improve this answer
    
Thanks Dr. Iorns for the quick and detailed reply, that are the references I was missing on many blogs and news sites explaining your initiative recently. I would put those references on your website, so it becomes clear it's esp. a major problem in medical sciences. – Hauser Nov 5 '12 at 22:04
    
This would also make a good answer to skeptics.stackexchange.com/questions/1973/… – matt_black Nov 6 '12 at 8:28
2  
Also raises questions about evidence-based medicine, if the evidence has been cooked: skeptics.stackexchange.com/questions/3111/… – mmr Nov 6 '12 at 22:07
1  
@mmr: I don't think that anyone is alleging widespread falsification or deliberate distortion of evidence (which the term "cooking" seems to imply). Rather the problem is a combination of systemic biases in science as a whole; pressure to produce positive results, post-hoc analysis and poor understanding of statistics often seem to combine to produce "statistically significant" results out of noise, while at the same time negative results tend to remain unpublished. – Paul Johnson Feb 24 at 18:08

There are several studies that document the problems of reproducibility, and thoroughly review the issue in indications of medicine, oncology, and neuroscience.

The two separate publications below detail how over 2/3 of landmark oncology studies were found not to be reproducible.

Asadullah: http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html

Glenn Begley / Lee Ellis: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html

share|improve this answer
    
Thanks to you too, someone informed both of you really quick ;) – Hauser Nov 5 '12 at 22:05
2  
To make this a better answer, could you summarise the content of the linked papers? – matt_black Nov 8 '12 at 23:05

Yes, and the problem exists across many areas of science

The problem of reproducibility is attracting a lot more attention. A recent Nature News Feature discusses the possible causes of bad results in the literature and many examples where they have been exposed. Its headline sums up the problem:

Humans are remarkably good at self-deception. But growing concern about reproducibility is driving many researchers to seek ways to fight their own worst instincts.

One huge problem is that results that sound plausible get a lot less scrutiny than unusual results so wrong but plausible results get a free pass on error checking far too often.

The article summarises a range of different reproducibility experiments and the results suggest that the majority of results across many fields are not reproducible (it is worth reading the article for all the links to these studies).

Failure to understand our own biases has helped to create a crisis of confidence about the reproducibility of published results, says statistician John Ioannidis, co-director of the Meta-Research Innovation Center at Stanford University in Palo Alto, California. The issue goes well beyond cases of fraud. Earlier this year, a large project that attempted to replicate 100 psychology studies managed to reproduce only slightly more than one-third. In 2012, researchers at biotechnology firm Amgen in Thousand Oaks, California, reported that they could replicate only 6 out of 53 landmark studies in oncology and haematology. And in 2009, Ioannidis and his colleagues described how they had been able to fully reproduce only 2 out of 18 microarray-based gene-expression studies.

On a more positive note the article recommends a number of ways to de-bias scientific answers. Some of these are summarised in the picture below:

Image (c) Nature

So there are plenty of known reasons why bad results get published and there are a lot of them. But it is something that good scientists can fight.

share|improve this answer
1  
This doesn't actually answer the question. The real problem why bad results get published is that there are too many journals for the available pool of competent reviewers and too much "publish or perish" pressure on the researchers. The cognitive bias thing isn't as much of a problem as is being portrayed and is distracting attention from where the problem really lies. Scientists are already trained to deal with these biases, the most successful scientists are generally those who are good at doing so. – Dikran Marsupial Feb 24 at 12:18
2  
@DikranMarsupial You are right that the proliferation of journals is part of the problem, but you are wrong that scientists are good at avoiding these biases. Even good scientists screw up because of them (see the Andrew Gelman story in the Nature article). And the evidence says that a lot of crap is published. Moreover, scientists are incentivised to advance their careers by publishing and this is synergistic with poor quality peer review to generate bad, published results. – matt_black Feb 24 at 13:35
    
"Even good scientists screw up because of them" you are missing the point, yes we all have these biases and none of us is perfect. However scientists are already trained in avoiding them and are vastly better at avoiding them than the general public (e.g. compare public and scientific reactions to Prof. Salby's carbon cycle arguments). Sure top scientists still make errors, but part of the reason they are top scientists is that they are better at dealing with their biases than the rest. – Dikran Marsupial Feb 24 at 13:44
    
1  
@DikranMarsupial On that very topic, a defence of the original study and some more general criticism of the state of scientific evidence in psychology. – matt_black Mar 6 at 14:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.