When we think about peer review, we usually think of the traditional model of researchers doing a study and then submitting it to a journal where it is then critiqued. But there are other ways already being implemented. We talk to Dr Paul Whaley all about registered reports and systematic reviews and his vision of peer review.
As you know at Crastina we are exploring peer review and different ways of publishing, but before we get to that, I’m interested to hear more about systematic reviews and Registered Reports. Could you tell us what systematic reviews are because they are not something everyone will know about?
Systematic review is a way of answering questions using existing research. Say you’ve got lots of studies in a space, for example the effects of air pollution on health. Well, it may not be necessary to conduct a new primary study if the answer to the question that we would be investigating is already known. So, say we want to know the answer to “what effects does diesel particulates have on cancer risk?” You could, of course, do a new study. But that would be a waste if we already had enough existing studies that allowed us to come to a reasonably certain answer. What separates systematic reviews from other types of review is that we’ve got a set of techniques and methods that we use for minimizing bias when we’re using existing evidence to answer research questions. For instance, we do things like make sure that our searches are comprehensive. So that we find all of the potentially relevant studies and not just a few. We also make sure the validity of these studies by assessing them for potential systematic error or bias. This makes sure that error in the studies we are pooling together in our systematic review aren’t transmitted through to the findings of our review. Systematic review is still a relatively new way of reviewing the literature. It wasn’t until the late 1980s that a set of rules were formalised to make sure that reviews could be done systematically, i.e. in a transparent, reproducible and unbiased way. I rate them as probably one of the most important health innovations: in the 19th Century we had vaccination; shortly after we had antiseptics; in the early 20th Century we discovered antibiotics; and in the late 20th Century we had systematic review. I think systematic review is one of the most important innovations in generating and understanding health evidence that we’ve developed, because it allows us to distinguish reliably between effective treatments and ineffective ones, and we can now use it to look at the effects of the environment on health, to find what studies are needed and to consolidate the knowledge we already have.
At Environment International, where you were appointed the first specialist editor of systematic reviews in the field of environmental health, you take a novel approach to peer review by implementing “Registered Reports”. Could you explain to us this method of peer review and the advantages of this over traditional peer review?
Registered reports are a two-stage peer-review process where we peer-review the methods of a study before it has been conducted, and we provide a guarantee that so long as the final study follows the methods that we’ve reviewed, it will be published regardless of its results. So the condition for publication is not how interesting the results are (like many publications) but whether or not you conformed to your planned methods. The point of this is to reduce publication bias – that instead of studies being published because their results are exciting, they’re being published because their methods are robust. Peer reviewing methods before researchers conduct a piece of research does two things. It helps improve the study design because you can catch issues before it’s too late, since obviously once you’ve done the study you can’t change how you did it. And it also means that it reduces the opportunity for expectation bias to distort study results – because you can’t so easily choose your methods based around the results you are getting. I mean, most researchers don’t do this sort of thing on purpose but it’s very difficult, if you have certain expectations about what you’re going, to not let those expectations influence you design decisions. So, Registered Reports should reduce publication bias and improve the robustness of methods because you’re getting peer review before you’ve done you research.
Registered Reports should reduce publication bias and improve the robustness of methods because you’re getting peer review before you’ve done you research.
In your journal, you use an enhanced peer review system. So, as I understand it, two topic experts and two method experts. Does this improve peer-review, and do the reviewers remain the same throughout the two-stage peer review?
We think it does, though I should emphasise that we haven’t got much empirical data. Ideally we would do a controlled trial to see if four reviewers is better than three; until then, it is just my feeling, as an editor, with my experience of systematic reviews where we’ve had four peer reviewers, they get more reviewer comments. It picks up more issues and I think it results in a better peer review process.
We try to keep the same reviewers for both stages of the process, but, of course, people’s availability changes a lot. We might desire to have people review the systematic review who’ve already reviewed the protocol, but it’s not always possible. When this happens, we have to instruct the reviewers that they’re not reviewing the methods because the methods are already agreed – they’re getting the results and interpretation.
You are already making changes in how the peer review process takes place, so how do you see the future of peer review? What would be your ideal?
I’m in favour of anything that improves peer review, though we don’t have a lot of solid empirical evidence about what actually is effectively improving peer review. I think there’s a certain element of social contract to being an academic. I think if you are publishing papers in journals and you’re benefiting from the peer review process then you should contribute to the peer review process. But there’s a lot to untangle and I haven’t personally untangled very much of it. It’s going to be an ongoing narrative for some time I think.
There’s an interesting initiative related to registered reports called the “Peer Community In Registered Report” (PCI-RR) (https://peercommunityin.org/). It was set up quite recently. This partnership is an attempt to make peer review of methods more accessible. So, anybody with a stage one study that they’re planning to do can in theory submit their study plan to PCI-RR and get it peer-reviewed so it doesn’t have to go to a journal. It just goes to this this kind of like self-designated community of people who are saying “okay I’ll peer review outside of the journal framework” and then once you’ve had that approved then any journal that endorses the PCIRR review process should in theory mean that they wouldn’t need to peer review that manuscript themselves as long as the study is in the scope of journal. But it also means, for the authors, they can demonstrate they’ve had something peer-reviewed even if they haven’t been to a journal with them. They can have their pre-print online and say “yeah it’s stage one approved” and then they can take the final results of the study back to the PCI-RR group. They’ll peer review the results and findings. Then if that’s accepted as well then they can say they’ve got a recommendation to publish from PCI-RR and put that on their pre-print. You can actually have something then that is peer-reviewed and a pre-print not even in a journal but should, in theory, carry the same weight as a paper that has been published in a journal. It’s the peer review that really counts.
In terms of my ideal? I’d like to see some research into how to make peer review work really well. What works? What doesn’t? I think two-stage review is essential so we should be doing all research, pretty much all empirical research, on the two-stage model – so you review methods before you publish the study. I think there should be much less publishing. I think it would just be easier if we were expected to publish less and we were not measured on citation counts and things, and then we had some sensible measures of research quality. I’m not just saying that because I’m lazy and I want to publish four papers a year, but I even think four papers is a lot because they’re really hard work – and they should be hard work.
I’d like to see some research into how to make peer review work really well. What works? What doesn’t?
I just feel like that would be good if we could compensate peer reviews for their time. That would be fantastic! You’re not just relying on the social contract. I think editors should be paid to manage the peer review process. I’m very fortunate because I work for Elsevier and they pay me an honorarium for my time. Not all publishers do that. A lot of editors I know work for free. I mean if we talk about paying peer reviewers we should definitely be paying editors. That social contract, right? I think we can develop tools that make peer review more transparent and comprehensive. That would be good! Understanding when to use tools and how to use them would be really, really useful. I think open peer review is pretty important, but it needs to be handled quite carefully. Coming up with ways of making publishing cheaper. That would be good. Get rid of assessing people on impact factor and on the journal they published in, but some better measure of quality of research.
I just feel like that would be good if we could compensate peer reviews for their time. That would be fantastic! You’re not just relying on the social contract.
Finally, I’d like to talk about your research. I see you’re looking at artificial intelligence (AI) and machine learning (ML) for automated evidence synthesis. It sounds like a perfect use for those tools. Could you tell us more about it?
I’ve been running a consultancy for 10/11 years um and started a PhD part time in 2013 because I wanted to move into academia. The work I do is mainly around trying to understand how to improve publishing standards in secondary research and then, as in so far as I can, apply that in primary research. But being a systematic reviewer and developing systematic methods, that’s where I mainly work. At the moment, we’re working on developing empirical evidence for how well systematic reviews are conducted in the environmental health field. I’m also a professional editor and much of my time is careful thinking about editorial processes and developing these processes and tools.
If you would like to learn more about systematic reviews and registered reports, check out Paul’s website: https://www.whaleyresearch.uk/
- Another Way? – Interview with Dr Paul Whaley - July 29, 2021
- Is Sci-Hub Science’s Napster? - May 28, 2021
- Music and Education - September 7, 2020
- Interview with Glenn Wolkenfeld – the man behind sciencemusicvideos - July 21, 2020
- Interview with Lucy Poley – Tips to make great graphical abstracts - July 1, 2020
- Hana Ayoob – Talks science comedy, podcasts, and art - April 22, 2020
- Let’s talk about cancer, separating facts and untruths – an interview with Lisa Whittaker of Tenovus - December 16, 2019