4

A former editor of the prestigious scientific journal the BMJ has recently argued that the peer review process for scientific publications is broken and should be abandoned. He argues (my emphasis):

Richard Smith, who edited the BMJ between 1991 and 2004, told the Royal Society’s Future of Scholarly Scientific Communication conference on 20 April that there was no evidence that pre-publication peer review improved papers or detected errors or fraud.

Referring to John Ioannidis’ famous 2005 paper “Why most published research findings are false”, Dr Smith said “most of what is published in journals is just plain wrong or nonsense”. He added that an experiment carried out during his time at the BMJ had seen eight errors introduced into a 600-word paper that was sent out to 300 reviewers.

“No one found more than five [errors]; the median was two and 20 per cent didn’t spot any,” he said. “If peer review was a drug it would never get on the market because we have lots of evidence of its adverse effects and don’t have evidence of its benefit.”

Is he right that there is no evidence that peer review improves the quality or reduces the errors of scientific publications?

Update and clarification

I'm guilty of expressing this question is a way that missed the point it should be addressing. The current question above allows the interpretation that any improvement ever in the history of scientific publishing proves the question to be false. That wasn't my intent.

There is a serious argument (perhaps expressed too strongly by Smith) that peer review does, on balance, a rotten job. We expect it to eliminate serious statistical errors, mistakes, evidence that doesn't actually back key conclusions and other related mistakes. He argues it does a poor job at that process and that there are alternatives.

The way I hoped people might try to answer this question would be to address the question of the underlying evidence not to aim for the rhetorically satisfying hit of showing that any proof a paper was ever improved is a satisfactory answer. Smith shows that the majority of know errors are missed. He shows that most reviewers miss most errors and some miss all the errors.

What I hoped was that answers might address whether Smith's evidence is any good and whether other people have similar or contradictory evidence. That evidence will be the same whether or not we all agree on the detailed purpose of peer review. And it is an important question for this site as it puts a lot of weight on the "peer-reviewed" scientific literature.

I'm adding this as an update as I suspect that rewriting the whole question would just annoy those who have already answered.

13

Pre-publication peer review is not supposed to catch all errors or mistakes. It's a relatively superficial review (they aren't going to generally perform a rigorous study to validate or invalidate the findings in the pre-publication step) to see if basic, remedial standards of professionalism have been followed before publication.

Peer review is not currently designed to detect deception, nor does it guarantee the validity of research findings. It should, however, identify flaws in the design, presentation, analysis and interpretation of science and provide prompt, detailed, constructive criticism to improve research. (Lee, Bero, Nature (2006))

Nature (2006): Ethics - Increasing accountability; peer review debate

The actual post-publication process is the part that really is "peer review" - this is where methods, data and conclusions are rigorously examined and tested and conclusions about the validity of the papers is really done. No scientific finding is considered valid or accepted until other scientists have examined, critiqued and attempted to replicate results, using the same methods as the original experiments/studies, or different methods.

The process of peer review does not end after a paper completes the pre-publication peer review process. After being put to press, or having been digitally published, the process of peer review continues a publications are read.

Wikipedia: Scholarly peer review

By his own examination, 80% caught errors in the submitted papers, so in 80% of the cases the paper would have undergone some correction or not published as it was, and that's before the more rigorous post-publication review process. That means, in 80% of those cases, the paper would have been improved in the pre-publication process. That's quite different than "no evidence" of improvement. Mind you, catching some, but not all, errors is an improvement.

Sounds like his claims don't match with his data. Perhaps if he would have submitted his speech as a paper, it would not have been published in a peer-reviewed journal.

  • 6
    The "refereed" term also highlights the correct points from this response - the point of pre-publication peer review is not to detect fraud or even to guarantee correctness. It is to ensure that the study meets some minimal standards - only obvious flaws are generally found. The greater community ultimately will judge correctness. – KAI Apr 26 '17 at 23:56
  • 4
    @matt_black I don't think you have read the answer properly, I read PoloHoleSet's answer as saying that pre-publication review is not supposed to detect errors, that is the purpose of post-publication review. The answer also points out that the editors own study refutes his claim as the reviewers did spot some of the errors, and thus it does improve the papers and/or detects some of the errors. For some papers, the error can't be reliably detected without repeating the study, which the reviewer simply doesn't have the time or resources to do. – Dikran Marsupial Apr 27 '17 at 6:37
  • 2
    @matt_black scholarly peer review is not supposed to be a guarantee of finding all errors in a paper. Whether they should be picked up or whether they are obvious is a matter of opinion, however a peer review that identifies any error will have improved the paper, and hence the results refute the hypothesis as posed. – Dikran Marsupial Apr 27 '17 at 8:39
  • 2
    @matt_black you need to apologise to PolpHoleSet then as there is nothing in their answer that uses a misleading statistic. As I pointed out the original claim doesn't require that peer review catches all errors. I disagree that your intepretation of the claim is more reasonable, it isn't it is more extreme as you appear to want review to detect all errors, rather than just "detect errors", which is what Smith actually wrote. If your last point were reasonable, we wouldn't have need of meta-analysis, which is very commonly used in medical journals. – Dikran Marsupial Apr 27 '17 at 9:04
  • 2
    @matt_black - it's not really a "clear piece of terminology" if it has many different meanings, depending on the context. Also, there's nothing misleading, since I explicitly point out that I'm not claiming that they are catching all errors. It would be more misleading to claim that catching some errors is "no improvement." I'm not redefining anything, I'm quoting what others say it is. – PoloHoleSet Apr 27 '17 at 13:35
10

While many agree that the current peer review system is not as effective as people would like it to be, it is not true that there is "no evidence that pre-publication peer review improved papers or detected errors or fraud". Authors overwhelmingly say that peer review improved the quality of their own last published paper.

In a 2009 survey of academics [1],

Ninety-one percent (±0.9%, p < 0.05, n = 4,037) of respondents agreed that the review process improved the quality of the last paper they published.

They further indicated which aspects of their last paper were improved by peer review:enter image description here

Similarly, in a 2008 survey of academics [2] supported by the Publishing Research Consortium, a group representing publishers and societies interested in research on scholarly communication,

the large majority of authors (around 90%) were clear that peer review had improved their own last published paper and a similar proportion agreed with the more general statement ‘peer review improves the quality of the published paper’.

These respondents were also asked to identify the specific area of improvement in their own last published paper due to peer review:

enter image description here

(Notably, 64% said that peer review of their last published paper had identified scientific errors.)

This is not evidence based on a blinded controlled study, so not the highest standard of evidence - but it is evidence nonetheless.

There is, however, general agreement that peer review is not as effective at detecting fraud and plagiarism as academics think it should be.

In [1],

Eighty-one percent (±1.2%, n = 4,037, p < 0.05) expect peer review to detect plagiarism, but just 38% (±1.5%, p < 0.05) feel that the current system is able to do this. Similarly, 79% (±1.3%, p < 0.05, n = 4,037) would like peer review to detect fraud, compared with 33% (±1.5%, p < 0.05, n = 4,037) who feel it is successful in this aspect.

And in [2], far fewer researchers surveyed said that peer review is effective at detecting fraud and plagiarism, than said it improves paper quality:

enter image description here

Whether peer review is "broken", and whether it is worth the costs associated with it, are matters of opinion. (Researchers' opinions on this are also included in the two surveys cited below.)


References

[1] Mulligan, Adrian, Louise Hall, and Ellen Raphael. "Peer review in a changing world: An international study measuring the attitudes of researchers." Journal of the American Society for Information Science and Technology 64, no. 1 (2013): 132-161. DOI: 10.1002/asi.22798

[2] Ware, Mark. "Peer review: benefits, perceptions and alternatives." Publishing Research Consortium 4 (2008). URL

  • 1
    Whether authors like peer review isn't the same as whether peer review does a good job for science as a whole. Indeed one major criticism is that the process is nepotistic allowing bad publications from small communities of peers who agree with each other but often rejecting interesting but controversial but well conducted work from outsiders. But, apart from that, a useful answer. – matt_black Apr 27 '17 at 9:03
  • 5
    @matt_black The question doesn't ask if peer review is good for science (that's a matter of opinion), it asks if peer review improves paper quality. "Author perception of paper quality" is a valid quality metric (though certainly, not the only one). – ff524 Apr 27 '17 at 9:12

Not the answer you're looking for? Browse other questions tagged .