Classical peer review: Where is the evidence? (Joshua Nicholson, October 2014)

Has classical peer review really been proven to work?  In the first monthly Crastina Column Joshua Nicholson shares his doubts with us. He is one of the creators of the open access journal The Winnower – ”DIY Scientific Publishing” which is is ”founded on the principle that all ideas in science should be openly discussed and debated”.

Joshua Nicholson is a graduate student at Virginia Tech where he studying the role of the karyotype in cancer initiation and progression.   is one of the co-founders of The Winnover.

Joshua Nicholson is a graduate student at Virginia Tech where he studies the role of the karyotype in cancer initiation and progression. As a reply to the eternal question ”Does the publication system make sense?” he and his colleagues decided to found a open access journal: The Winnower – DIY Scientific Publishing.

Science is said to be evidence-based—that is one needs evidence to support or refute a hypothesis. It has been proposed and defended that classical peer review is a suitable check for the quality of scientific research. I ask here, as a scientist, where is the evidence?

Classical peer review is the process in which 2-4 peers (colleagues and competitors) review a paper looking for clarity, errors, fraud, and many times even “potential impact.” The process occurs away from the public eye, is single-blinded (i.e. the reviewers are anonymous to authors), and can take weeks, even years to complete (Bornmann and Daniel, 2010).

Overall, it is a slow process that is not generally open for evaluation by the scientific community. Is it effective? When editors at various journals artificially introduced errors into studies and sent them for review, the majority went undetected.

  • Editors at the British Medical Journal found on average that only 2 out of 9 major errors were detected (Schroter et al., 2008).
  • At JAMA only 2 out of 8 errors were noticed (Godlee et al., 1998).
  • At the Annals of Emergency Medicine, similarly, 68% of reviewers did not realize the conclusions were not supported by the evidence (Baxt et al., 1998).

Thus, the majority of errors go undetected by classical peer review, supporting the conclusion that it is not effective.

What’s the solution? Open post-publication peer review (ie open commentary after publication with the ability to revise and update publications). Disagree? Then write a review! But careful, because that’s open post-publication peer review working.

Joshua Nicholson, PhD student at Virginia Tech
Email: jnicholson at thewinnower.com
Interview at the blog Open@VT

References

  • Baxt, W.G., Waeckerle, J.F., Berlin, J.A., Callaham, M.L., 1998. Who Reviews the Reviewers? Feasibility of Using a Fictitious Manuscript to Evaluate Peer Reviewer Performance. Annals of Emergency Medicine 32, 310-317.
  • Bornmann, L., Daniel, H.-D., 2010. How long is the peer review process for journal manuscripts? A case study on Angewandte Chemie International Edition. Chimia (Aarau) 64, 72-77.
  • Godlee, F., Gale, C.R., Martyn, C.N., 1998. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. JAMA 280, 237-240.
  • Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., Smith, R., 2008. What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine 101, 507-514.
8 replies
  1. Olle Bergman
    Olle Bergman says:

    As in all situations when a traditional and a new doctrine are challenging each other, I come to think about the lyrics to Dylan’s ”The Times They Are A-changin’”, released in 1964.

    ”And admit that the waters
    Around you have grown
    And accept it that soon
    You’ll be drenched to the bone
    [—]
    Then you better start swimmin’
    Or you’ll sink like a stone”

    http://www.metrolyrics.com/the-times-they-are-achangin-lyrics-bob-dylan.html

    Reply
  2. Nathalie
    Nathalie says:

    I think (and sincerely hope) that the majority of researchers today are aware that the peer review system is flawed, but so far no viable option has emerged.

    If the peer review system was to fall out of use, the scientific journals will eventually also become superfluous. With the alternative being free online publishing, it’s going to be harder to find relevant articles when you search for information.
    I’m also concerned that any system that requires the masses to determine what is good science will benefit mainly those that are already established within the particular field simply because they will generate more traffic…

    Don’t get me wrong, I appreciate any attempt to improve the system- I just think we still have a long way to go!

    Reply
  3. jenny
    jenny says:

    Intresting, wondering about which these errors were… But what would be an alternative? So many manuscripts are sent in, and honestly myself I did feel several times that the manuscript improved with the comments I got by the reviewers.

    Reply
  4. Josh N.
    Josh N. says:

    Peer review can work in some instances, however, even when it does work it doesn’t at the same time. That is even if one or two or three journals reject a paper, there will be a 4th journal that accepts it. All the negative reviews (justified or not) from previous journals will never see the light of day. It would be much better for all papers to enter a repository and then receive open reviews from verified researchers–this is what we are hoping to create. Open peer review, by definition, must occur post-publication, since “rejected” and “approved” articles must be published.

    The way peer review works now, it is if anything, stopping different viewpoints and ideas from entering the scientific literature. It is, as referenced above, not stopping errors.

    Hope you’ll consider signing up at thewinnower.com and participating in some reviews.

    Reply
  5. Álvaro
    Álvaro says:

    I think that all the opinions above this one reflect that the scientific community is aware of the flaws of the peer review system; and therefore where we should focus so that this scenario may change.

    In my opinion, I do not really know how much impact can an open peer review system have in the future; since I am reluctant to think that the community would take as valid an article which has not been validated by the current peer review system. Maybe the self entitled reviewer does have some interest on that paper being successfully reviewed, or more importantly, viceversa.

    Besides, maybe not everyone has the means to reproduce the experiments referred on such paper, and then it is easier for fake or fraudulent investigations to make it through. It is sad, but it has already happened, we all know about Hwang Woo-suk (http://www.nature.com/news/2009/091026/full/4611181a.html).

    I am definitely not a supporter of the current peer review system, but I am still skeptical to believe that open peer reviewing will be the solution; hope it is!

    Reply
  6. Par J. Hoglund
    Par J. Hoglund says:

    So when is an article “done” – when is it frozen in time. One view is at the timepoint you submit with the potential for major revisions. We tend to look at thing quite statically. Another view would be of looking at articles as more ongoing projects, similar to some degree to which a wikipedia article is developing. A way forward (fresh thought) would be to wikipedize the publication system. It would need a central backbone system and everything else is distributed and anyone can comment on the papers and the authors can continue to respond and to update the article after the suggestion from reviewers.

    Just some fast thought during the lunch break 🙂

    Reply
  7. Norbert
    Norbert says:

    As others has already reffered to it it’s not perfect but we don’t have another system yet which works better. Just like democracy…
    “No one pretends that democracy is perfect or all wise. Indeed, it has been said that democracy is
    the worst form of government, except for all the others that have been tried from time to time.”
    Winston Churchill, House of Commons speech on Nov. 11, 1947

    Complex systems where each entity has a purpose, not necessarily same as the communities purpose, will always be flawed by selfish acts from one side, and rigidity by the rules which are imposed by the system.

    However if we look at the bypassing errors, against what the system is invented, it gets even worrier.

    Reply

Trackbacks & Pingbacks

  1. […] you for returning to Crastina to update us on your work Josh. Since you’ve last spoken to us at Crastina you have accomplished a lot in terms of completing your PhD but also creating platforms that improve […]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.