Classical peer review: Where is the evidence? (Joshua Nicholson, October 2014)
Has classical peer review really been proven to work? In the first monthly Crastina Column Joshua Nicholson shares his doubts with us. He is one of the creators of the open access journal The Winnower – ”DIY Scientific Publishing” which is is ”founded on the principle that all ideas in science should be openly discussed and debated”.
Joshua Nicholson is a graduate student at Virginia Tech where he studies the role of the karyotype in cancer initiation and progression. As a reply to the eternal question ”Does the publication system make sense?” he and his colleagues decided to found a open access journal: The Winnower – DIY Scientific Publishing.
Science is said to be evidence-based—that is one needs evidence to support or refute a hypothesis. It has been proposed and defended that classical peer review is a suitable check for the quality of scientific research. I ask here, as a scientist, where is the evidence?
Classical peer review is the process in which 2-4 peers (colleagues and competitors) review a paper looking for clarity, errors, fraud, and many times even “potential impact.” The process occurs away from the public eye, is single-blinded (i.e. the reviewers are anonymous to authors), and can take weeks, even years to complete (Bornmann and Daniel, 2010).
Overall, it is a slow process that is not generally open for evaluation by the scientific community. Is it effective? When editors at various journals artificially introduced errors into studies and sent them for review, the majority went undetected.
- Editors at the British Medical Journal found on average that only 2 out of 9 major errors were detected (Schroter et al., 2008).
- At JAMA only 2 out of 8 errors were noticed (Godlee et al., 1998).
- At the Annals of Emergency Medicine, similarly, 68% of reviewers did not realize the conclusions were not supported by the evidence (Baxt et al., 1998).
Thus, the majority of errors go undetected by classical peer review, supporting the conclusion that it is not effective.
What’s the solution? Open post-publication peer review (ie open commentary after publication with the ability to revise and update publications). Disagree? Then write a review! But careful, because that’s open post-publication peer review working.
Joshua Nicholson, PhD student at Virginia Tech
Email: jnicholson at thewinnower.com
Interview at the blog Open@VT
References
- Baxt, W.G., Waeckerle, J.F., Berlin, J.A., Callaham, M.L., 1998. Who Reviews the Reviewers? Feasibility of Using a Fictitious Manuscript to Evaluate Peer Reviewer Performance. Annals of Emergency Medicine 32, 310-317.
- Bornmann, L., Daniel, H.-D., 2010. How long is the peer review process for journal manuscripts? A case study on Angewandte Chemie International Edition. Chimia (Aarau) 64, 72-77.
- Godlee, F., Gale, C.R., Martyn, C.N., 1998. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. JAMA 280, 237-240.
- Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., Smith, R., 2008. What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine 101, 507-514.
- Claire Price of Crastina receives outreach award from Royal Society of Biology - October 25, 2020
- Agile Science student project at Brussels Engineering School ECAM: “We can’t wait to try it again!” - August 28, 2020
- Create an infographic in the Lifeology SciArt Infographic Challenge - June 16, 2020
- Adam Ruben – The scientist that teaches undergraduate students comedy - March 27, 2020
- Sam Gregson, Bad Boy of Science: “Comedy helps to bridge the gap” - March 10, 2020
- The Coolest Science Merchandise of 2019 - December 16, 2019
- Science Media Centre (UK) offers guide on dealing with online harassment in academia - November 26, 2019
- Agile project management taught to students and researchers at Karolinska Institutet - September 20, 2019
- Stefan Jansson: Improve your credibility! (Crastina Column, September 2019) - September 6, 2019
- The People’s Poet: Silke Kramprich, tech communicator - August 31, 2019
As in all situations when a traditional and a new doctrine are challenging each other, I come to think about the lyrics to Dylan’s ”The Times They Are A-changin’”, released in 1964.
”And admit that the waters
Around you have grown
And accept it that soon
You’ll be drenched to the bone
[—]
Then you better start swimmin’
Or you’ll sink like a stone”
http://www.metrolyrics.com/the-times-they-are-achangin-lyrics-bob-dylan.html
I think (and sincerely hope) that the majority of researchers today are aware that the peer review system is flawed, but so far no viable option has emerged.
If the peer review system was to fall out of use, the scientific journals will eventually also become superfluous. With the alternative being free online publishing, it’s going to be harder to find relevant articles when you search for information.
I’m also concerned that any system that requires the masses to determine what is good science will benefit mainly those that are already established within the particular field simply because they will generate more traffic…
Don’t get me wrong, I appreciate any attempt to improve the system- I just think we still have a long way to go!
Intresting, wondering about which these errors were… But what would be an alternative? So many manuscripts are sent in, and honestly myself I did feel several times that the manuscript improved with the comments I got by the reviewers.
Peer review can work in some instances, however, even when it does work it doesn’t at the same time. That is even if one or two or three journals reject a paper, there will be a 4th journal that accepts it. All the negative reviews (justified or not) from previous journals will never see the light of day. It would be much better for all papers to enter a repository and then receive open reviews from verified researchers–this is what we are hoping to create. Open peer review, by definition, must occur post-publication, since “rejected” and “approved” articles must be published.
The way peer review works now, it is if anything, stopping different viewpoints and ideas from entering the scientific literature. It is, as referenced above, not stopping errors.
Hope you’ll consider signing up at thewinnower.com and participating in some reviews.
I think that all the opinions above this one reflect that the scientific community is aware of the flaws of the peer review system; and therefore where we should focus so that this scenario may change.
In my opinion, I do not really know how much impact can an open peer review system have in the future; since I am reluctant to think that the community would take as valid an article which has not been validated by the current peer review system. Maybe the self entitled reviewer does have some interest on that paper being successfully reviewed, or more importantly, viceversa.
Besides, maybe not everyone has the means to reproduce the experiments referred on such paper, and then it is easier for fake or fraudulent investigations to make it through. It is sad, but it has already happened, we all know about Hwang Woo-suk (http://www.nature.com/news/2009/091026/full/4611181a.html).
I am definitely not a supporter of the current peer review system, but I am still skeptical to believe that open peer reviewing will be the solution; hope it is!
So when is an article “done” – when is it frozen in time. One view is at the timepoint you submit with the potential for major revisions. We tend to look at thing quite statically. Another view would be of looking at articles as more ongoing projects, similar to some degree to which a wikipedia article is developing. A way forward (fresh thought) would be to wikipedize the publication system. It would need a central backbone system and everything else is distributed and anyone can comment on the papers and the authors can continue to respond and to update the article after the suggestion from reviewers.
Just some fast thought during the lunch break 🙂
As others has already reffered to it it’s not perfect but we don’t have another system yet which works better. Just like democracy…
“No one pretends that democracy is perfect or all wise. Indeed, it has been said that democracy is
the worst form of government, except for all the others that have been tried from time to time.”
Winston Churchill, House of Commons speech on Nov. 11, 1947
Complex systems where each entity has a purpose, not necessarily same as the communities purpose, will always be flawed by selfish acts from one side, and rigidity by the rules which are imposed by the system.
However if we look at the bypassing errors, against what the system is invented, it gets even worrier.