Are most published results relying on measures of statistical significance likely to be false?

Question

This article, linked to by Glen, claims most published results are false. The argument is based on an analysis of how papers relying on statistical significance analyse and report their statistical results and is bolstered by related arguments about publication bias.

An intuitive way to understand the possibility of publication bias is to note that insignificant results (e.g. this drug is no better than placebo) don't make great publications but apparently significant results (like MMR vaccine causes autism) will get you headlines in the popular press long after other evidence has refuted your conclusion. And trawling random data for significant results will throw up many results with p-values significant at the 5% level. Publishing such results while ignoring the insignificant results of the trawl is a poor way to determine truth but a good way to get publication kudos. Some modern authors believe this is endemic not just in social science but in medical epidemiology.

So it is important to know: are most published results likely to be wrong?

Answer 1

The "most epidemiology studies are wrong", perhaps best advanced by John P. A. Ioannidis in Why Most Published Research Findings Are False both extremely common and somewhat flawed, though there are absolutely some worthwhile points in his article.

Perhaps the one that amuses me the most is how readily it's been embraced conceptually, even though its argument is that...perhaps it shouldn't be.

Goodman and Greenland outline some problems with the paper here: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0040168

Mainly, the flaw is in how Ioannidis structures his argument, such that the odds are stacked against certain types of studies, and the evidentiary value of effect measures and actual p-values, rather than just a binary "Is/is not >0.05", are discounted.

There are other examples of supposedly "wrong" findings later disproved (this usually goes in the form of an observational epidemiology study finding one thing, and a subsequent RCT finding something different) that can be traced back to the studies asking different things. An examplar paper might be: Hernán MA, Alonso A, Logan R, Grodstein F, Michels KB, Willett WC, Manson JE, Robins JM. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease (with discussion). Epidemiology 2008; 19:766-779.

What I take from all this is the following:

1. Significance testing in and of itself is not proof. There's far more value in providing the p-value, and even more value in providing an actual estimate of effect.
2. No study can provide singular, definitive proof of something. Not an observational study, and not an RCT (which are far from free of bias). Nor should any study claim to.
3. The question of "wrong" is an odd one in science. A more useful question would be "are most published results likely to advance their field?". Because a wrong study that provides fodder for further analysis, new methods and more thoughtful science is still profoundly useful.

Answer 2

I made that point referring to this post in which I question the conclusion of spanking causing aggressive behavior: Is spanking an effective form of discipline for children?.

There is a publication bias, especially in the social sciences (can't publish insignificant effects). When you can run controlled experiments, as in chemistry, physics, or clinical trials, the results are much more likely to be true. However in observational studies the tempetation is to go from association to causation, which in repeated studies the effect probably does not hold.

More on the subject:

http://www.johndcook.com/blog/2011/01/17/scientific-results-fading-over-time/