8
votes

It's a quite popular claim among scientific communities that real science is that which is published in peer-reviewed journals.

However, good science is about theories that can be tested, evidence that can be reproduced and verified. Peer-reviewing essentially seems to be like a popularity contest ( a model that is present from skeptics.SE to reality TV shows).

So, what research has been done on whether peer-reviewed journals leads to better science? About problems/drawbacks of the system?

10
  • 3
    What kind of evidence would you find acceptable for this question?
    – Sklivvz
    May 9, 2011 at 5:50
  • 1
    As is the case with most of science, negative evidence is easier to test. So studies on whether there is bias in the peer-reviewing system. Or how reproducible or true articles in journals are.
    – apoorv020
    May 9, 2011 at 7:40
  • I agree, however--are you willing to trust a peer-reviewed study? It's circular logic :-)
    – Sklivvz
    May 9, 2011 at 8:06
  • If somebody trusts all peer-reviewed studies, and one study claims that they are not trustworthy then it's a contradiction. If I take them to be equivalent to non-peer-reviewed studies, then a particular study claiming drawbacks in the peer-review system is not a fallacy.
    – apoorv020
    May 9, 2011 at 8:09
  • "Are all peer-reviewed studies true?" is a question that's very different from "Are peer-reviewed studies more likely to be true?". As it stands there no single well defined claim in the Question that can be answered directly.
    – Christian
    May 9, 2011 at 11:43

1 Answer 1

Reset to default
9
votes

There has been a lot of work on this subject, most of which has even been peer reviewed! Even 30+ years ago, there were studies confirming confirmatory bias in peer review:

...we are left with little to defend [the peer review process] other than tradition.

("Confirmatory bias" is one's tendancy to agree with conclusions that agree with one's own preconceived notions and hypotheses.)

That study took 75 journal reviewers (I believe they were all from the cognitive sciences community) and presented them with papers to review. The catch was that the papers were all identical in their experimental setup and data, however, they varied in their conclusions of the results. The majority of the reviewers agreed with the conclusions that were similar to their own theoretical perspective.

Peer review of research grant proposals isn't much better; this study from around the same time period concluded that being awarded a peer-reviewed research grant falls mostly to chance:

The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.

From a philosophical perspective peer-review can be very effective, however, it relies mostly on the quality of the reviewer:

Peer review can be performed successfully only if those involved have a clear idea as to its fundamental purpose. Most authors of articles on the subject assume that the purpose of peer review is quality control. This is an inadequate answer. The fundamental purpose of peer review in the biomedical sciences must be consistent with that of medicine itself, to cure sometimes, to relieve often, to comfort always. Peer review must therefore aim to facilitate the introduction into medicine of improved ways of curing, relieving, and comforting patients. The fulfillment of this aim requires both quality control and the encouragement of innovation. If an appropriate balance between the two is lost, then peer review will fail to fulfill its purpose.

This more recent meta-study aggregated the results of 19 studies on peer review (having pruned away 116 other studies that did not meet their criteria). They found that each study had widely different conclusions, many of which may not be generalizable. They conclude that

Editorial peer review, although widely used, is largely untested and its effects are uncertain.

Some studies did show that peer review is effectual in improving the readability of articles.

So, in summary, there is a lot of evidence to support the fact that the effect of peer review is often indistinguishable from chance. It has been perpetuated due to tradition; until very recently, the dissemination of scholarly information was technologically and economically limited to distribution via printed journals, and peer review provided a means for limiting the size of those expensive journals.

For some more amusing reading on this subject, I suggest you check out the opinions of Doron Zeilberger (e.g., this one).

4
  • How does this correlate to the different sciences?
    – Sklivvz
    May 9, 2011 at 15:08
  • @Sklivvz: I believe the last meta-study aggregated results from the various sciences and medicine. The "confirmatory bias" result also likely generalizes to all disciplines. I couldn't find anything else that specifically analyzed peer review over multiple disciplines. Perhaps it is still an area of open research? If so, we should all write a proposal for a research grant to investigate it! If the studies I linked above are correct, our likelihood of being funded will be at least as good as random! ;-)
    – ESultanik
    May 9, 2011 at 17:15
  • Excellent answer. Thanks very much. I was afraid this was going to go unanswered.
    – apoorv020
    May 9, 2011 at 18:27
  • 1
    There is a major move on in particle physics to "blind" the analysis as often as possible. By blind we mean apply some reversible transformation to the "almost raw" data seen by the analysis team, so that they can't (consciously or unconsciously) twiddle the cuts and corrections to get the expected answer. This practice is increasingly common. No word yet on how much it helps.
    – dmckee --- ex-moderator kitten
    May 9, 2011 at 22:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .