What is… Peer Review?

Click here for a 3-minute video explaining the peer review process.
Image credit: Pixabay

You may have heard that science uses a process called “peer review” to determine which articles are worthy of publication in a scientific journal. But what does that actually mean, and how does it work? In this post, I describe the peer review process, why it is important for science, and some criticisms of the process.

 What is peer review, and why is it important?

Peer review can be defined as “a process of subjecting an author’s scholarly work, research, or ideas to the scrutiny of others who are experts in the same field.” (1) Peer review is important because it helps ensure that high-quality research is published in scientific journals. Journals vary in the quality of research they will publish, but the most well-respected journals strive to only accept papers that answer meaningful questions and draw accurate conclusions based on sound methodology. Peer review helps identify which articles meet the standards of good science and often provide authors with suggestions on ways to improve their papers before publication.

The Peer Review Process

Process of peer review. Modified from Kelly, Sadeghieh, & Adeli (5).

Although the peer review process varies slightly across scientific fields and journals, the typical process goes as follows: First, a scientist (or team of scientists) completes a research study and writes a paper outlining the purpose of the study, relevant background, method, results, and conclusions of their work. The scientist/s submit their paper to an academic journal (usually through an online submission system). The editor of the journal will then evaluate whether the paper fits the goals of the journal and if it is worth sending the paper out for peer review. Many papers are rejected at this stage.

If the paper passes this initial evaluation, the editor sends the paper out to experts in the field for peer review. The editor oversees the peer review process to make sure it is fair and effective. For example, they try to avoid potential conflicts of interest in the process (e.g., enlisting a reviewer who has a personal relationship with the author of the paper). They may also provide their own comments about the paper that need to be addressed by the study author/s.

Reviewers (sometimes called “referees”) typically receive a limited amount of time (e.g., one month) to write a brief report in which they evaluate the significance of the research, quality of the study design, appropriateness of the method and conclusions drawn from the results, and whether there are any errors in the writing. They also provide a recommendation to the editor about what to do with the paper. Based on these reports, the editor makes a decision to either (1) reject the paper, (2) accept the paper, or (3) offer the author/s an opportunity to revise the paper based on the reviewers’ comments. The editor typically writes a brief report of their own summarizing the reviews and then sends this decision letter, along with the peer reviewers’ reports (“reviews”), to the scientist/s who submitted the paper. They may or may not be able to see the names of the reviewers who evaluated their paper, depending on journal policy or the reviewers’ decision about whether or not to remain anonymous.

 If the paper is accepted, it is published in the journal along with other papers and made available to readers. If a paper is rejected (which is common, particularly at prestigious journals), the author/s can decide to submit it to another journal. Typically, a paper can only be “under review” with one journal at a time.

If a revision is requested, the scientist/s can decide whether to revise and resubmit their paper (often with a letter explaining how they addressed prior reviews) to the journal at a later time for another round of review. Because scientific papers often undergo multiple rounds of rejections, revisions, and additional data collection before publication, the entire process can take months or even years.

Who conducts peer reviews?

Academic writing is typically very specialized, so peer reviews are conducted by scientists with expertise relevant to the content of the paper submitted. Journals often try to pull from a diverse pool of reviewers to allow for multiple perspectives. Scientists may choose to review papers for multiple reasons – for example, to stay up to date on the latest research, to show they are committed to their field, and to help them get promoted.

Post-publication Peer Review

The process doesn’t stop once a paper is published! As the publication is read by a wider audience, scientists (and other “peers”) will continue to critique the research. Readers may send letters to the editor of the journal, email the authors directly, or provide comments in online communities. They may also conduct follow-up research to demonstrate that an idea from a published paper was wrong, or try to replicate the results in new settings.

Problems with Peer Review

According to a study by the Publishing Research Consortium, the average reviewer writes eight reviews per year (2). Though this may not sound like a lot, consider that it takes the average reviewer nine hours to write a review, and that reviewers are not paid for their time. Given the millions of papers that are submitted to journals each year, it can be challenging to find good reviewers with relevant expertise who can write a quality review in a timely manner. Without good reviews, no doubt that low-quality papers end up published even in prestigious scientific journals.

Another issue is that the peer review process is not immune to bias (3). For example, reviewers may be particularly critical of ideas that counter “mainstream” theories or conclusions that contradict their own views, which can stunt scientific innovation and progress. Aside from content, characteristics of the author, such as the prestige of their affiliated institution, nationality, language, and gender have also been found to influence reviewers and editors in their decision-making process. 

Finally, peer review is not equipped to detect all errors in every paper. Data fraud (i.e., making up data) and plagiarism are particularly difficult to detect. In rare cases, papers are removed (“retracted”) from the journal after publication because their credibility is questioned some time in the future (such as the infamous paper that suggested a link between MMR vaccines and autism, despite inadequate data; 4). These failures in the process are particularly detrimental because they can lead to distrust in the scientific process by the general public.

Conclusion

Ideally, the peer review process serves as a quality-control measure to ensure that only high-quality research ends up being read by the scientific community and the general public. Despite its flaws and deficiencies, the majority of scientists agree that peer review is the best system we have to select the best papers for scientific publication (2)

References:

1.  https://en.wikipedia.org/wiki/Scholarly_peer_review. Accessed 25 November 2019.
2.  PRC Peer review survey 2015. Publishing Research Consortium.
3.  Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2-17.
4.  Rao, T. S., & Andrade, C. (2011). The MMR vaccine and autism: Sensation, refutation, retraction, and fraud. Indian Journal of Psychiatry, 53(2), 95-96.
5.  Kelly, J., Sadeghieh, T., & Adeli, K. (2014). Peer review in scientific publications: benefits, critiques, & a survival guide. EJIFCC, 25(3), 227.