Open Peer Review
What is Open Peer Review, and how is it different from other review models?
Peer review is a pillar of scientific communication, the mechanism we rely on to ensure that published research is thoroughly vetted and scientifically valid. For that reason, we tend to think of peer review as a monolith–iconic, stable, and consistent. In fact, journals use many different forms and applications of peer review, often in parallel.
What does it mean to practice Open Peer Review?
That is sort of a trick question…Open Peer Review has many different definitions. It is a general, catch-all term used to describe any peer review model in which aspects of the peer review process are made publicly available, either before or after publication.
For example, Open Peer Review models may include any of the following transparent practices, either alone or in combination:
- publishing peer review content
- open commenting from the wider community
- open discussion between authors, editors and reviewers
- open review before publication through preprints
- post-publication commenting
- sharing author or reviewer identities
- decoupling the peer review process from the publication process
At least 22 different peer review configurations were already in use across scientific publishing as of 2017 (Ross-Hellauer).
It’s important to treat each attribute of transparent review separately, since each can have a different impact on the author, reviewer, and reader experience. Here, we’ll be talking primarily about publishing peer review content, as well as touching lightly on signed review.
Published Peer Review History at PLOS
Published Peer Review History collects the correspondence exchanged during the peer review possess—including decision letters from each revision, complete with both editorial feedback and peer reviews, and the authors’ responses to reviewers—and makes it available alongside a published research article as a permanent part of the scientific record. PLOS authors can opt-in to publish their Peer Review History once their article is accepted for publication. If the reviewers have chosen to sign their review, their names will appear alongside their comments. If not, the reviews will appear anonymously.
Why is Open Peer Review important?
Depending on your discipline and the length of time you’ve worked in research, you may see Open Peer Review as industry standard, entirely new and novel, or somewhere inbetween.
Open Peer Review in one form or another has been part of the scholarly communications conversation since the 1980s (McGiffert, 1988). Both signed and published peer review have been practiced at The BMJ and BMJ family journals for over 20 years (van Rooyen 1998, 1999), which may partially account for their prevalence in medical and health-related fields. Public review is similarly accepted in computational fields―perhaps because many researchers have a background in coding, and are comfortable with the public commenting common on GitHub and similar code repositories.
In the life sciences, Open Peer Review is less established but is becoming increasingly normalized. For example, journals including Royal Society Open Science, Nature Communications, EMBO, eLife, and of course the PLOS journals, all offer forms of Open Peer Review.
No matter the focus of your research, the odds are good that you’ll be invited to participate in an Open Peer Review process during your career―if you haven’t already.
The value of published peer review
Proponents of published review argue that increasing transparency in the peer review process leads to a better understanding of published research, more constructive peer reviews, and well-deserved credit for reviewers.
✔ Enriching the scientific record
Published peer review helps contextualize research and gives readers the benefit of additional expert opinions. Letting readers see the questions reviewers raised and how the authors mitigated them gives insight into the limits of the study. Publishing peer reviews also reinforces the validity of the article by exposing the rigorous vetting process it has undergone prior to publication.
✔ Honoring reviews and reviewers
Choosing to publish peer reviews acknowledges the value of review, and the vital role that it plays in scientific communication by making it a permanent part of the scientific record. Offering reviewers the opportunity to sign is just one of several ways for them to claim credit for their work.
✔ Educational tools
Making peer review publicly available creates a database of examples for students to reference as they begin to participate in peer review.
✔ Quality of feedback
Research suggests reviews with the potential to be published tend to be at least as good as, and maybe slightly better than, reviews which remain private. Two BMJ studies on signed review (Van Rooyen, 2001) and impact of possible public posting (van Rooyen, 2010) found no difference in quality; however, other research found improvements in specific areas like constructive feedback, comments on methods, length of review, and substantiating evidence to support the comments (Kowalczuk, 2013; Walsh, 2000; Bornmann, 2012; Mehmani, 2016).
Concerns about published peer review
✘ Diminishing reviewer capacity
Just as telling reviewers their comments may be published can improve the quality of the review, some detractors argue that publishing reviews puts undue pressure on reviewers. For example, knowing that a peer review could be published may lead reviewers to spend extra time on spelling or grammar―aspects of the review which really don’t impact the science, but which can be time consuming, especially for researchers who primarily use other languages. That could contribute to a more demanding and stressful experience for reviewers, and less time for review, and more declined invitations.
On signed peer review
Revealing reviewer identities to authors helps to contextualize the feedback, providing insight into potential competing interests, and emphasizing once again the rigor, thoroughness, and appropriateness of the peer review assessment. Signing also gives reviewers an opportunity to claim academic credit for their contributions to the study. That potential only increases when signing operates in tandem with published review.
At the same time, revealing reviewer identities is by far the most common reason stakeholders object to openness in peer review. Many in the research community worry about the threat of negative career consequences for critical reviewers, especially for junior researchers who are dependent on senior scientists for opportunities and advancement.
Harassment, bullying, and blackballing do occur in science–and even anonymous review is not necessarily protection. According to some studies, authors can guess the identities of their reviewers in as many as 10% (van Rooyen, 1998) to 32% (Justice, 1998) of cases. Even if the authors are wrong about the identities of their reviewers, that doesn’t necessarily mean that career damage won’t occur―for someone. These concerns are valid, and those of us on the periphery of research, like publishers, funders, and administrators, often can’t know the political and social circumstances in which researchers are operating. For that reason, it’s important that individual reviewers have the flexibility to decide whether to sign their name to a review in each specific case.
Meanwhile, journals, funders and institutions can have a positive impact by explicitly naming retribution against peer reviewers as misconduct with consequences (Bastian, 2018), and by making the whole process more transparent and open to public scrutiny, thereby reducing opportunities to engage in negative behaviors.
Have your say
How do I complete an “Open,” signed, or published peer review?
First, decide where you stand. Take some time to review the research, talk to colleagues you respect, and consider whether you want to sign your reviews as a general principle. Remember, just because you have a personal philosophy doesn’t mean there won’t be occasional exceptions–but at least this way you’ll feel confident about your choices and the reasons behind them.
The practical aspects of completing a peer review remain the same, regardless of the review model a particular journal might happen to employ. There are three key stages to completing a peer review, Open or otherwise.
- Responding to invitations. Accept an invitation to review only if you have the necessary time and expertise, and can provide an objective review. Journals will let you know what peer review model they practice before you agree to review, usually in the invitation letter itself, or on their website. Read more.
- Reading the manuscript. Before you read the manuscript, make sure that you understand the journal’s criteria, so you know what to watch for as you make your assessment. Read the manuscript through once for a general understanding, and a second time taking notes as you go. Pay special attention to the research question, methods, and conclusions, and review the figures and tables in conjunction with the results. Read more.
- Writing the Review. Begin with a summary of the research and your overall impression, before moving on to discuss specific areas for improvement and any other points. There is no need to do the authors’ work for them by suggesting line edits, specific manuscripts to cite, or experiments to do. Just let the editors know where you had questions or concerns. Read more.
Citations:
Bastian, H. (2018, March 22). Signing Critical Peer Reviews & the Fear of Retaliation: What Should we do? [Blog post]. Absolutely Maybe.
Bolam, P. (2017, September 14). Transparent Review at the European Journal of Neuroscience: Experiences One Year On, [Blog post].
Bornmann, L., Wolf, M. & Daniel, H.D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91(3): 843-856.
Justice A.C., Cho M.K., Winker M.A., Berlin J.A., Rennie D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial, JAMA. 280(3):240-2.
Kowalczuk, M.K , Dudbridge, F. , Nanda, S., Harriman, S.L., & Moylan, E.C. (2013). A comparison of the quality of reviewer reports from author-suggested reviewers and editor-suggested reviewers in journals operating on open or closed peer review models. F1000Research.
McGiffert M. (1988). Is Justice Blind? An Inquiry into Peer Review. Scholarly Publishing, 20(1): 43-48.
Mehmani, B. (2016, September 22). Is Open Peer Review the Way Forward? [Blog].
Pulverer, B. (2010). A transparent black box. The EMBO Journal, 29(23):3891-3892.
Ross-Hellauer, T. (2017). What is open peer review? A systematic review [version 2; peer review: 4 approved]. F1000Research, 6:588
van Rooyen, S., Godlee, F., Evans, S., Smith, R., Black, N. (1998) Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA, 280(3):234-7.
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999, Jan 2). Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ, 318(7175):23-7.
van Rooyen, S., Delamothe, T. & Evans, S.J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ, 341:5729.
Walsh, E. Rooney, M. & Wilinson, G. (2000). Open peer review: a randomised controlled trial. British Journal of Psychiatry, 176:47-51.