peer reviewed articles on literature review

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Credit: Getty

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

WENTING ZHAO: Be focused and avoid jargon

Assistant professor of chemical and biomedical engineering, Nanyang Technological University, Singapore.

When I was a research student, review writing improved my understanding of the history of my field. I also learnt about unmet challenges in the field that triggered ideas.

For example, while writing my first review 1 as a PhD student, I was frustrated by how poorly we understood how cells actively sense, interact with and adapt to nanoparticles used in drug delivery. This experience motivated me to study how the surface properties of nanoparticles can be modified to enhance biological sensing. When I transitioned to my postdoctoral research, this question led me to discover the role of cell-membrane curvature, which led to publications and my current research focus. I wouldn’t have started in this area without writing that review.

Collection: Careers toolkit

A common problem for students writing their first reviews is being overly ambitious. When I wrote mine, I imagined producing a comprehensive summary of every single type of nanomaterial used in biological applications. It ended up becoming a colossal piece of work, with too many papers discussed and without a clear way to categorize them. We published the work in the end, but decided to limit the discussion strictly to nanoparticles for biological sensing, rather than covering how different nanomaterials are used in biology.

My advice to students is to accept that a review is unlike a textbook: it should offer a more focused discussion, and it’s OK to skip some topics so that you do not distract your readers. Students should also consider editorial deadlines, especially for invited reviews: make sure that the review’s scope is not so extensive that it delays the writing.

A good review should also avoid jargon and explain the basic concepts for someone who is new to the field. Although I trained as an engineer, I’m interested in biology, and my research is about developing nanomaterials to manipulate proteins at the cell membrane and how this can affect ageing and cancer. As an ‘outsider’, the reviews that I find most useful for these biological topics are those that speak to me in accessible scientific language.

Bozhi Tian likes to get a variety of perspectives into a review. Credit: Aleksander Prominski

BOZHI TIAN: Have a process and develop your style

Associate professor of chemistry, University of Chicago, Illinois.

In my lab, we start by asking: what is the purpose of this review? My reasons for writing one can include the chance to contribute insights to the scientific community and identify opportunities for my research. I also see review writing as a way to train early-career researchers in soft skills such as project management and leadership. This is especially true for lead authors, because they will learn to work with their co-authors to integrate the various sections into a piece with smooth transitions and no overlaps.

After we have identified the need and purpose of a review article, I will form a team from the researchers in my lab. I try to include students with different areas of expertise, because it is useful to get a variety of perspectives. For example, in the review ‘An atlas of nano-enabled neural interfaces’ 2 , we had authors with backgrounds in biophysics, neuroengineering, neurobiology and materials sciences focusing on different sections of the review.

After this, I will discuss an outline with my team. We go through multiple iterations to make sure that we have scanned the literature sufficiently and do not repeat discussions that have appeared in other reviews. It is also important that the outline is not decided by me alone: students often have fresh ideas that they can bring to the table. Once this is done, we proceed with the writing.

I often remind my students to imagine themselves as ‘artists of science’ and encourage them to develop how they write and present information. Adding more words isn’t always the best way: for example, I enjoy using tables to summarize research progress and suggest future research trajectories. I’ve also considered including short videos in our review papers to highlight key aspects of the work. I think this can increase readership and accessibility because these videos can be easily shared on social-media platforms.

ANKITA ANIRBAN: Timeliness and figures make a huge difference

Editor, Nature Reviews Physics .

One of my roles as a journal editor is to evaluate proposals for reviews. The best proposals are timely and clearly explain why readers should pay attention to the proposed topic.

It is not enough for a review to be a summary of the latest growth in the literature: the most interesting reviews instead provide a discussion about disagreements in the field.

Careers Collection: Publishing

Scientists often centre the story of their primary research papers around their figures — but when it comes to reviews, figures often take a secondary role. In my opinion, review figures are more important than most people think. One of my favourite review-style articles 3 presents a plot bringing together data from multiple research papers (many of which directly contradict each other). This is then used to identify broad trends and suggest underlying mechanisms that could explain all of the different conclusions.

An important role of a review article is to introduce researchers to a field. For this, schematic figures can be useful to illustrate the science being discussed, in much the same way as the first slide of a talk should. That is why, at Nature Reviews, we have in-house illustrators to assist authors. However, simplicity is key, and even without support from professional illustrators, researchers can still make use of many free drawing tools to enhance the value of their review figures.

Yoojin Choi recommends that researchers be open to critiques when writing reviews. Credit: Yoojin Choi

YOOJIN CHOI: Stay updated and be open to suggestions

Research assistant professor, Korea Advanced Institute of Science and Technology, Daejeon.

I started writing the review ‘Biosynthesis of inorganic nanomaterials using microbial cells and bacteriophages’ 4 as a PhD student in 2018. It took me one year to write the first draft because I was working on the review alongside my PhD research and mostly on my own, with support from my adviser. It took a further year to complete the processes of peer review, revision and publication. During this time, many new papers and even competing reviews were published. To provide the most up-to-date and original review, I had to stay abreast of the literature. In my case, I made use of Google Scholar, which I set to send me daily updates of relevant literature based on key words.

Through my review-writing process, I also learnt to be more open to critiques to enhance the value and increase the readership of my work. Initially, my review was focused only on using microbial cells such as bacteria to produce nanomaterials, which was the subject of my PhD research. Bacteria such as these are known as biofactories: that is, organisms that produce biological material which can be modified to produce useful materials, such as magnetic nanoparticles for drug-delivery purposes.

Synchronized editing: the future of collaborative writing

However, when the first peer-review report came back, all three reviewers suggested expanding the review to cover another type of biofactory: bacteriophages. These are essentially viruses that infect bacteria, and they can also produce nanomaterials.

The feedback eventually led me to include a discussion of the differences between the various biofactories (bacteriophages, bacteria, fungi and microalgae) and their advantages and disadvantages. This turned out to be a great addition because it made the review more comprehensive.

Writing the review also led me to an idea about using nanomaterial-modified microorganisms to produce chemicals, which I’m still researching now.

PAULA MARTIN-GONZALEZ: Make good use of technology

PhD student, University of Cambridge, UK.

Just before the coronavirus lockdown, my PhD adviser and I decided to write a literature review discussing the integration of medical imaging with genomics to improve ovarian cancer management.

As I was researching the review, I noticed a trend in which some papers were consistently being cited by many other papers in the field. It was clear to me that those papers must be important, but as a new member of the field of integrated cancer biology, it was difficult to immediately find and read all of these ‘seminal papers’.

That was when I decided to code a small application to make my literature research more efficient. Using my code, users can enter a query, such as ‘ovarian cancer, computer tomography, radiomics’, and the application searches for all relevant literature archived in databases such as PubMed that feature these key words.

The code then identifies the relevant papers and creates a citation graph of all the references cited in the results of the search. The software highlights papers that have many citation relationships with other papers in the search, and could therefore be called seminal papers.

My code has substantially improved how I organize papers and has informed me of key publications and discoveries in my research field: something that would have taken more time and experience in the field otherwise. After I shared my code on GitHub, I received feedback that it can be daunting for researchers who are not used to coding. Consequently, I am hoping to build a more user-friendly interface in a form of a web page, akin to PubMed or Google Scholar, where users can simply input their queries to generate citation graphs.

Tools and techniques

Most reference managers on the market offer similar capabilities when it comes to providing a Microsoft Word plug-in and producing different citation styles. But depending on your working preferences, some might be more suitable than others.

Reference managers

Here is a comparison of the more popular collaborative writing tools, but there are other options, including Fidus Writer, Manuscript.io, Authorea and Stencila.

Collaborative writing tools

*Markdown and LaTex are code-based formatting languages favoured by physicists, mathematicians and computer scientists who code on a regular basis, and less popular in other disciplines such as biology and chemistry.

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

Why you should perform a premortem on your research

Career Column 24 JUL 24

Science must protect thinking time in a world of instant communication

Editorial 24 JUL 24

A stroke derailed my academic career. Now I’m getting back on track

Career Column 23 JUL 24

So you got a null result. Will anyone publish it?

News Feature 24 JUL 24

Hijacked journals are still a threat — here’s what publishers can do about them

Nature Index 23 JUL 24

China–US research collaborations are in decline — this is bad news for everyone

News 19 JUL 24

Postdoctoral Research Scientist

Computational Postdoctoral Research Scientist in Neurodegenerative Disease

New York City, New York (US)

Columbia University Department of Pathology

Associate or Senior Editor (Chemical Engineering), Nature Sustainability

Associate or Senior Editor (Chemical engineering), Nature Sustainability Locations: New York, Philadelphia, Berlin, or Heidelberg Closing date: Aug...

Springer Nature Ltd

Postdoctoral Research Scientist - Mouse Model

The Stavros Niarchos Foundation Center for Precision Psychiatry & Mental Health is seeking a Postdoctoral Research Scientist for the Mouse Model.

Columbia University Irving Medical Center / Vagelos College of Physicians and Surgeons

Postdoctoral Research Scientist - IPSC Program

The Stavros Niarchos Foundation Center for Precision Psychiatry & Mental Health is seeking a Postdoctoral Research Scientist - IPSC Program.

Postdoctoral Research Scientist - Xenotransplantation

The Stavros Niarchos Foundation Center for Precision Psychiatry & Mental Health is seeking a Postdoctoral Research Scientist -Xentotransplantation.

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Literature reviews as independent studies: guidelines for academic practice

• Original Paper
• Open access
• Published: 14 October 2022
• Volume 16 , pages 2577–2595, ( 2022 )

Cite this article

You have full access to this open access article

• Sascha Kraus   ORCID: orcid.org/0000-0003-4886-7482 1 , 2 ,
• Matthias Breier 3 ,
• Weng Marc Lim 4 , 8 , 22 ,
• Marina Dabić 5 , 6 ,
• Satish Kumar 7 , 8 ,
• Dominik Kanbach 9 , 10 ,
• Debmalya Mukherjee 11 ,
• Vincenzo Corvello 12 ,
• Juan Piñeiro-Chousa 13 ,
• Eric Liguori 14 ,
• Daniel Palacios-Marqués 15 ,
• Francesco Schiavone 16 , 17 ,
• Alberto Ferraris 18 , 21 ,
• Cristina Fernandes 19 , 20 &
• João J. Ferreira 19  

83k Accesses

329 Citations

4 Altmetric

Explore all metrics

Review articles or literature reviews are a critical part of scientific research. While numerous guides on literature reviews exist, these are often limited to the philosophy of review procedures, protocols, and nomenclatures, triggering non-parsimonious reporting and confusion due to overlapping similarities. To address the aforementioned limitations, we adopt a pragmatic approach to demystify and shape the academic practice of conducting literature reviews. We concentrate on the types, focuses, considerations, methods, and contributions of literature reviews as independent, standalone studies. As such, our article serves as an overview that scholars can rely upon to navigate the fundamental elements of literature reviews as standalone and independent studies, without getting entangled in the complexities of review procedures, protocols, and nomenclatures.

Similar content being viewed by others

Literature Reviews: An Overview of Systematic, Integrated, and Scoping Reviews

On being ‘systematic’ in literature reviews

Overview of the Integrative Review

Avoid common mistakes on your manuscript.

1 Introduction

A literature review – or a review article – is “a study that analyzes and synthesizes an existing body of literature by identifying, challenging, and advancing the building blocks of a theory through an examination of a body (or several bodies) of prior work (Post et al. 2020 , p. 352). Literature reviews as standalone pieces of work may allow researchers to enhance their understanding of prior work in their field, enabling them to more easily identify gaps in the body of literature and potential avenues for future research. More importantly, review articles may challenge established assumptions and norms of a given field or topic, recognize critical problems and factual errors, and stimulate future scientific conversations around that topic. Literature reviews Footnote 1 come in many different formats and purposes:

Some review articles conduct a critical evaluation of the literature, whereas others elect to adopt a more exploratory and descriptive approach.

Some reviews examine data, methodologies, and findings, whereas others look at constructs, themes, and theories.

Some reviews provide summaries by holistically synthesizing the existing research on a topic, whereas others adopt an integrative approach by assessing related and interdisciplinary work.

The number of review articles published as independent or standalone studies has been increasing over time. According to Scopus (i.e., search database ), reviews (i.e., document type ) were first published in journals (i.e., source type ) as independent studies in 1945, and they subsequently appeared in three digits yearly from the late 1980s to the late 1990s, four digits yearly from the early 2000s to the late 2010s, and five digits in the year 2021 (Fig.  1 ). This increase is indicative that reviewers and editors in business and management research alike see value and purpose in review articles to such a level that they are now commonly accepted as independent, standalone studies. This development is also reflected in the fact that some academic journals exclusively publish review articles (e.g., the Academy of Management Annals , or the  International Journal of Management Reviews ), and journals publishing in various fields often have special issues dedicated to literature reviews on certain topic areas (e.g., the Journal of Management and the Journal of International Business Studies ).

Full-year publication trend of review articles on Scopus (1945–2021)

One of the most important prerequisites of a high-quality review article is that the work follows an established methodology, systematically selects and analyzes articles, and periodically covers the field to identify latest developments (Snyder 2019 ). Additionally, it needs to be reproducible, well-evidenced, and transparent, resulting in a sample inclusive of all relevant and appropriate studies (Gusenbauer and Haddaway 2020; Hansen et al. 2021 ). This observation is in line with Palmatier et al. ( 2018 ), who state that review articles provide an important synthesis of findings and perspectives in a given body of knowledge. Snyder ( 2019 ) also reaffirmed this rationale, pointing out that review articles have the power to answer research questions beyond that which can be achieved in a single study. Ultimately, readers of review articles stand to gain a one-stop, state-of-the-art synthesis (Lim et al. 2022a ; Popli et al. 2022) that encapsulates critical insights through the process of re-interpreting, re-organizing, and re-connecting a body knowledge (Fan et al. 2022 ).

There are many reasons to conduct review articles. Kraus et al. ( 2020 ) explicitly mention the benefits of conducting systematic reviews by declaring that they often represent the first step in the context of larger research projects, such as doctoral dissertations. When carrying out work of this kind, it is important that a holistic overview of the current state of literature is achieved and embedded into a proper synthesis. This allows researchers to pinpoint relevant research gaps and adequately fit future conceptual or empirical studies into the state of the academic discussion (Kraus et al., 2021 ). A review article as an independent or standalone study is a viable option for any academic – especially young scholars, such as doctoral candidates – who wishes to delve into a specific topic for which a (recent) review article is not available.

The process of conducting a review article can be challenging, especially for novice scholars (Boell and Cecez-Kecmanovic 2015 ). Therefore, it is not surprising that numerous guides have been written in an attempt to improve the quality of review studies and support emerging scholars in their endeavors to have their work published. These guides for conducting review articles span a variety of academic fields, such as engineering education (Borrego et al. 2014 ), health sciences (Cajal et al. 2020 ), psychology (Laher and Hassem 2020 ), supply chain management (Durach et al. 2017 ), or business and entrepreneurship (Kraus et al. 2020 ; Tranfield et al. 2003 ) – the latter were among the first scholars to recognize the need to educate business/management scholars on the roles of review studies in assembling, ascertaining, and assessing the intellectual territory of a specific knowledge domain. Furthermore, they shed light on the stages (i.e., planning the review, conducting the review, reporting, and dissemination) and phases (i.e., identifying the need for a review, preparation of a proposal for a review, development of a review protocol, identification of research, selection of studies, study quality assessment, data extraction and monitoring progress, data synthesis, the report and recommendations, and getting evidence into practice) of conducting a systematic review. Other scholars have either adapted and/or developed new procedures (Kraus et al. 2020 ; Snyder 2019 ) or established review protocols such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram (Moher et al. 2015 ). The latter provides a checklist that improves transparency and reproducibility, thus reducing questionable research practices. The declarative and procedural knowledge of a checklist allows users to derive value from (and, in some cases, produce) methodological literature reviews.

Two distinct and critical gaps or issues provide impetus for our article. First, while the endeavors of the named scholars are undoubtedly valuable contributions, they often encourage other scholars to explain the methodology of their review studies in a non-parsimonious way ( 1st issue ). This can become problematic if this information distracts and deprives scholars from providing richer review findings, particularly in instances in which publication outlets impose a strict page and/or word limit. More often than not, the early parts (i.e., stages/phases, such as needs, aims, and scope) of these procedures or protocols are explained in the introduction, but they tend to be reiterated in the methodology section due to the prescription of these procedures or protocols. Other parts of these procedures or protocols could also be reported more parsimoniously, for example, by filtering out documents, given that scientific databases (such as Scopus or Web of Science ) have since been upgraded to allow scholars to select and implement filtering criteria when conducting a search (i.e., criterion-by-criterion filtering may no longer be necessary). More often than not, the procedures or protocols of review studies can be signposted (e.g., bracket labeling) and disclosed in a sharp and succinct manner while maintaining transparency and replicability.

Other guides have been written to introduce review nomenclatures (i.e., names/naming) and their equivalent philosophical underpinnings. Palmatier et al. ( 2018 ) introduced three clearly but broadly defined nomenclatures of literature reviews as independent studies: domain-based reviews, theory-based reviews, and method-based reviews. However, such review nomenclatures can be confusing due to their overlapping similarities ( 2nd issue ). For example, Lim et al. ( 2022a ) highlighted their observation that the review nomenclatures associated with domain-based reviews could also be used for theory-based and method-based reviews.

The two aforementioned issues – i.e., the lack of a parsimonious understanding and the reporting of the review methodology , and the confusion emerging from review nomenclatures – are inarguably the unintended outcomes of diving into an advanced (i.e., higher level) understanding of literature review procedures, protocols, and nomenclatures from a philosophical perspective (i.e., underpinnings) without a foundational (i.e., basic level) understanding of the fundamental (i.e., core) elements of literature reviews from a pragmatic perspective. Our article aims to shed light on these issues and hopes to provide clarity for future scholarly endeavors.

Having a foundational understanding of literature reviews as independent studies is (i) necessary when addressing the aforementioned issues; (ii) important in reconciling and scaffolding our understanding, and (iii) relevant and timely due to the proliferation of literature reviews as independent studies. To contribute a solution toward addressing this gap , we aim to demystify review articles as independent studies from a pragmatic standpoint (i.e., practicality). To do so, we deliberately (i) move away from review procedures, protocols, and nomenclatures, and (ii) invest our attention in developing a parsimonious, scaffolded understanding of the fundamental elements (i.e., types, focuses, considerations, methods, and contributions) of review articles as independent studies.

Three contributions distinguish our article. It is worth noting that pragmatic guides (i.e., foundational knowledge), such as the present one, are not at odds with extant philosophical guides (i.e., advanced knowledge), but rather they complement them. Having a foundational knowledge of the fundamental elements of literature reviews as independent studies is valuable , as it can help scholars to (i) gain a good grasp of the fundamental elements of literature reviews as independent studies ( 1st contribution ), and (ii) mindfully adopt or adapt existing review procedures, protocols, and nomenclatures to better suit the circumstances of their reviews (e.g., choosing and developing a well-defined review nomenclature, and choosing and reporting on review considerations and steps more parsimoniously) ( 2nd contribution ). Therefore, this pragmatic guide serves as (iii) a foundational article (i.e., preparatory understanding) for literature reviews as independent studies ( 3rd contribution ). Following this, extant guides using a philosophical approach (i.e., advanced understanding) could be relied upon to make informed review decisions (e.g., adoption, adaptation) in response to the conventions of extant review procedures, protocols, and nomenclatures (Fig.  2 ).

Foundational and advanced understanding of literature reviews as independent studies

2 Fundamental elements of literature reviews as independent studies

A foundational understanding of literature reviews as independent studies can be acquired through the appreciation of five fundamental elements – i.e., types, focuses, considerations, methods, and contributions – which are illustrated in Fig.  3 and summarized in the following sections.

Fundamental elements of literature reviews as independent studies

There are two types of literature reviews as independent studies: systematic literature reviews ( SLRs ) and non-systematic literature reviews ( non-SLRs ). It is important to recognize that SLRs and non-SLRs are not review nomenclatures (i.e., names/naming) but rather review types (i.e., classifications).

In particular, SLRs are reviews carried out in a systematic way using an adopted or adapted procedure or protocol to guide data curation and analysis, thus enabling transparent disclosure and replicability (Lim et al. 2022a ; Kraus et al. 2020 ). Therefore, any review nomenclature guided by a systematic methodology is essentially an SLR. The origin of this type of literature review can be traced back to the evidence-based medicine movement in the early 1990s, with the objective being to overcome the issue of inconclusive findings in studies for medical treatments (Boell and Cecez-Kecmanovic 2015 ).

In contrast, non-SLRs are reviews conducted without any systematic procedure or protocol; instead, they weave together relevant literature based on the critical evaluations and (subjective) choices of the author(s) through a process of discovery and critique (e.g., pointing out contradictions and questioning assertions or beliefs); they are shaped by the exposure, expertise, and experience (i.e., the “3Es” in judgement calls) of the author(s). Therefore, non-SLRs are essentially critical reviews of the literature (Lim and Weissmann 2021 ).

2.2 Focuses

Unlike Palmatier et al. ( 2018 ) who considered domain-based reviews, theory-based reviews, and method-based reviews as review nomenclatures, we consider domain , theory , and method as three substantive focuses that can take center stage in literature reviews as independent studies. This is in line with our attempt to move away from review nomenclatures when providing a foundational understanding of literature reviews as independent studies.

A review that is domain-focused can examine: (i) a  concept (e.g., customer engagement; Lim et al. 2022b ; digital transformation; Kraus et al. 2021 ; home sharing; Lim et al. 2021 ; sharing economy; Lim 2020 ), (ii) a context (e.g., India; Mukherjee et al. 2022a ), (iii) a discipline (e.g., entrepreneurship; Ferreira et al. 2015 ; international business; Ghauri et al. 2021 ), (iv) a field (e.g., family business; Lahiri et al. 2020 ; Rovelli et al. 2021 ; female entrepreneurship; Ojong et al. 2021 ), or (v) an outlet (e.g., Journal of Business Research ; Donthu et al. 2020 ; Management International Review ; Mukherjee et al. 2021 ; Review of Managerial Science ; Mas-Tur et al. 2020 ), which typically offer broad, overarching insights.

Domain-focused hybrids , such as the between-domain hybrid (e.g., concept-discipline hybrid, such as digital transformation in business and management; Kraus et al. 2022 ; religion in business and entrepreneurship; Kumar et al. 2022a ; personality traits in entrepreneurship; Salmony and Kanbach 2022 ; and policy implications in HR and OB research; Aguinis et al., 2022 ) and the within-domain hybrid (e.g., the concept-concept hybrid, such as customer engagement and social media; Lim and Rasul 2022 ; and global business and organizational excellence; Lim 2022 ; and the discipline-discipline hybrid, such as neuromarketing; Lim 2018 ) are also common as they can provide finer-grained insights.

A review that is theory-focused can explore a standalone theory (e.g., theory of planned behavior; Duan and Jiang 2008 ), as well as a theory in conjunction with a domain , such as the concept-theory hybrid (e.g., behavioral control and theory of planned behavior; Lim and Weissmann 2021 ) and the theory-discipline hybrid (e.g., theory of planned behavior in hospitality, leisure, and tourism; Ulker-Demirel and Ciftci 2020 ), or a theory in conjunction with a method (e.g., theory of planned behavior and structural equation modeling).

A review that is method-focused can investigate a standalone method (e.g., structural equation modeling; Deng et al. 2018 ) or a method in conjunction with a domain , such as the method-discipline hybrid (e.g., fsQCA in business and management; Kumar et al. 2022b ).

2.3 Planning the review, critical considerations, and data collection

The considerations required for literature reviews as independent studies depend on their type: SLRs or non-SLRs.

For non-SLRs, scholars often rely on the 3Es (i.e., exposure, expertise, and experience) to provide a critical review of the literature. Scholars who embark on non-SLRs should be well versed with the literature they are dealing with. They should know the state of the literature (e.g., debatable, underexplored, and well-established knowledge areas) and how it needs to be deciphered (e.g., tenets and issues) and approached (e.g., reconciliation proposals and new pathways) to advance theory and practice. In this regard, non-SLRs follow a deductive reasoning approach, whereby scholars initially develop a set of coverage areas for reviewing a domain, theory, or method and subsequently draw on relevant literature to shed light and support scholarly contentions in each area.

For SLRs, scholars often rely on a set of criteria to provide a well-scoped (i.e., breadth and depth), structured (i.e., organized aspects), integrated (i.e., synthesized evidence) and interpreted/narrated (i.e., describing what has happened, how and why) systematic review of the literature. Footnote 2 In this regard, SLRs follow an inductive reasoning approach, whereby a set of criteria is established and implemented to develop a corpus of scholarly documents that scholars can review. They can then deliver a state-of-the-art overview, as well as a future agenda for a domain, theory, or method. Such criteria are often listed in philosophical guides on SLR procedures (e.g., Kraus et al. 2020 ; Snyder 2019 ) and protocols (e.g., PRISMA), and they may be adopted/adapted with justifications Footnote 3 . Based on their commonalities they can be summarized as follows:

Search database (e.g., “Scopus” and/or “Web of Science”) can be defined based on justified evidence (e.g., by the two being the largest scientific databases of scholarly articles that can provide on-demand bibliographic data or records; Pranckutė 2021 ). To avoid biased outcomes due to the scope covered by the selected database, researchers could utilize two or more different databases (Dabić et al. 2021 ).

Search keywords may be developed by reading scholarly documents and subsequently brainstorming with experts. The expanding number of databases, journals, periodicals, automated approaches, and semi-automated procedures that use text mining and machine learning can offer researchers the ability to source new, relevant research and forecast the citations of influential studies. This enables them to determine further relevant articles.

Boolean operators (e.g., AND, OR) should be strategically used in developing the  string   of search keywords (e.g., “engagement” AND “customer” OR “consumer” OR “business”). Furthermore, the correct and precise application of quotation marks is important but is very frequently sidestepped, resulting in incorrect selection processes and differentiated results.

Search period (e.g., between a specified period [e.g., 2000 to 2020] or up to the latest full year at the time or writing [e.g., up to 2021]) can be defined based on the justified scope of study (e.g., contemporary evolution versus historical trajectory).

Search field (e.g., “article title, abstract, keywords”) can be defined based on justified assumptions (e.g., it is assumed that the focus of relevant documents will be mentioned in the article title, abstract, and/or keywords).

Subject area (e.g., “business, management, and accounting”) can be defined based on justified principles (e.g., the focus of the review is on the marketing discipline, which is located under the “business, management, and accounting” subject area in Scopus).

Publication stage (e.g., “final”) can be defined based on justified grounds (e.g., enabling greater accuracy in replication).

Document type (e.g., “article” and/or “review”), which reflects the type of scientific/practical contributions (e.g., empirical, synthesis, thought), can be defined based on justified rationales (e.g., articles selected because they are peer-reviewed; editorials not selected because they are not peer-reviewed).

Source type (e.g., “journal”) can be defined based on justified reasons (e.g., journals selected because they publish finalized work; conference proceedings not selected because they are work in progress, and in business/management, they are usually not being considered as full-fledged “publications”).

Language (e.g., “English”) can be determined based on justified limitations (e.g., nowadays, there are not many reasons to use another language besides the academic lingua franca English). Different spellings should also be considered, as the literature may contain both American and British spelling variants (e.g., organization and organisation). Truncation and wildcards in searches are recommended to capture both sets of spellings. It is important to note that each database varies in its symbology.

Quality filtering (e.g., “A*” and “A” or “4*”, “4”, and “3”) can be defined based on justified motivations (e.g., the goal is to unpack the most originally and rigorously produced knowledge, which is the hallmark of premier journals, such as those ranked “A*” and “A” by the Australian Business Deans Council [ABDC] Journal Quality List [JQL] and rated “4*”, “4”, and “3” by the Chartered Association of Business Schools [CABS] Academic Journal Guide [AJG]).

Document relevance (i.e., within the focus of the review) can be defined based on justified judgement (e.g., for a review focusing on customer engagement, articles that mention customer engagement as a passing remark without actually investigating it would be excluded).

Others: Screening process should be accomplished by beginning with the deduction of duplicate results from other databases, tracked using abstract screening to exclude unfitting studies, and ending with the full-text screening of the remaining documents.

Others: Exclusion-inclusion criteria interpretation of the abstracts/articles is obligatory when deciding whether or not the articles dealt with the matter. This step could involve removing a huge percentage of initially recognized articles.

Others: Codebook building pertains to the development of a codebook of the main descriptors within a specific field. An inductive approach can be followed and, in this case, descriptors are not established beforehand. Instead, they are established through the analysis of the articles’ content. This procedure is made up of several stages: (i) the extraction of important content from titles, abstracts, and keywords; (ii) the classification of this content to form a reduced list of the core descriptors; and (iii) revising the codebook in iterations and combining similar categories, thus developing a short list of descriptors (López-Duarte et al. 2016 , p. 512; Dabić et al. 2015 ; Vlacic et al. 2021 ).

2.4 Methods

Various methods are used to analyze the pertinent literature. Often, scholars choose a method for corpus analysis before corpus curation. Knowing the analytical technique beforehand is useful, as it allows researchers to acquire and prepare the right data in the right format. This typically occurs when scholars have decided upon and justified pursuing a specific review nomenclature upfront (e.g., bibliometric reviews) based on the problem at hand (e.g., broad domain [outlet] with a large corpus [thousands of articles], such as a premier journal that has been publishing for decades) (Donthu et al. 2021 ). However, this may not be applicable in instances where (i) scholars do not curate a corpus of articles (non-SLRs), and (ii) scholars only know the size of the corpus of articles once that corpus is curated (SLRs). Therefore, scholars may wish to decide on a method of analyzing the literature depending on (i) whether they rely on a corpus of articles (i.e., yes or no), and (ii) the size of the corpus of articles that they rely on to review the literature (i.e., n  = 0 to ∞).

When analytical techniques (e.g., bibliometric analysis, critical analysis, meta-analysis) are decoupled from review nomenclatures (e.g., bibliometric reviews, critical reviews, meta-analytical reviews), we uncover a toolbox of the following methods for use when analyzing the literature:

Bibliometric analysis measures the literature and processes data by using algorithm, arithmetic, and statistics to analyze, explore, organize, and investigate large amounts of data. This enables scholars to identify and recognize potential “hidden patterns” that could help them during the literature review process. Bibliometrics allows scholars to objectively analyze a large corpus of articles (e.g., high hundreds or more) using quantitative techniques (Donthu et al. 2021 ). There are two overarching categories for bibliometric analysis: performance analysis and science mapping. Performance analysis enables scholars to assess the productivity (publication) and impact (citation) of the literature relating to a domain, method, or theory using various quantitative metrics (e.g., average citations per publication or year, h -index, g -index, i -index). Science mapping grants scholars the ability to map the literature in that domain, method, or theory based on bibliographic data (e.g., bibliographic coupling generates thematic clusters based on similarities in shared bibliographic data [e.g., references] among citing articles; co-citation analysis generates thematic clusters based on commonly cited articles; co-occurrence analysis generates thematic clusters based on bibliographic data [e.g., keywords] that commonly appear together; PageRank analysis generates thematic clusters based on articles that are commonly cited in highly cited articles; and topic modeling generates thematic clusters based on the natural language processing of bibliographic data [e.g., article title, abstract, and keywords]). Footnote 4 Given the advancement in algorithms and technology, reviews using bibliometric analysis are considered to be smart (Kraus et al. 2021 ) and technologically-empowered (Kumar et al. 2022b ) SLRs, in which a review has harnessed the benefits of (i) the machine learning of the bibliographic data of scholarly research from technologically-empowered scientific databases, and (ii) big data analytics involving various science mapping techniques (Kumar et al. 2022c ).

Content analysis allows scholars to analyze a small to medium corpus of articles (i.e., tens to low hundreds) using quantitative and qualitative techniques. From a quantitative perspective , scholars can objectively carry out a content analysis by quantifying a specific unit of analysis . A useful method of doing so involves adopting, adapting, or developing an organizing framework . For example, Lim et al. ( 2021 ) employed an organizing (ADO-TCM) framework to quantify content in academic literature based on: (i) the categories of knowledge; (ii) the relationships between antecedents, decisions, and outcomes; and (iii) the theories, contexts, and methods used to develop the understanding for (i) and (ii). The rapid evolution of software for content analysis allows scholars to carry out complex elaborations on the corpus of analyzed articles, so much so that the most recent software enables the semi-automatic development of an organizing framework (Ammirato et al. 2022 ). From a qualitative perspective , scholars can conduct a content analysis or, more specifically, a thematic analysis , by subjectively organizing the content into themes. For example, Creevey et al. ( 2022 ) reviewed the literature on social media and luxury, providing insights on five core themes (i.e., luxury brand strategy, luxury brand social media communications, luxury consumer attitudes and perceptions, engagement, and the influence of social media on brand performance-related outcomes) generated through a content (thematic) analysis. Systematic approaches for inductive concept development through qualitative research are similarly applied in literature reviews in an attempt to reduce the subjectivity of derived themes. Following the principles of the approach by Gioia et al. ( 2012 ), Korherr and Kanbach ( 2021 ) develop a taxonomy of human-related capabilities in big data analytics. Building on a sample of 75 studies for the literature review, 33 first-order concepts are identified. These are categorized into 15 second-order themes and are finally merged into five aggregate dimensions. Using the same procedure, Leemann and Kanbach ( 2022 ) identify 240 idiosyncratic dynamic capabilities in a sample of 34 studies for their literature review. They then categorize these into 19 dynamic sub-capabilities. The advancement of technology also makes it possible to conduct content analysis using computer assisted qualitative data analysis (CAQDA) software (e.g., ATLAS.ti, Nvivo, Quirkos) (Lim et al. 2022a ).

Critical analysis allows scholars to subjectively use their 3Es (i.e., exposure, expertise, and experience) to provide a critical evaluation of academic literature. This analysis is typically used in non-SLRs, and can be deployed in tandem with other analyses, such as bibliometric analysis and content analysis in SLRs, which are used to discuss consensual, contradictory, and underexplored areas of the literature. For SLRs, scholars are encouraged to engage in critical evaluations of the literature so that they can truly contribute to advancing theory and practice (Baker et al. 2022 ; Lim et al. 2022a ; Mukherjee et al. 2022b ).

Meta-analysis allows scholars to objectively establish a quantitative estimate of commonly studied relationships in the literature (Grewal et al. 2018 ). This analysis is typically employed in SLRs intending to reconcile a myriad of relationships (Lim et al. 2022a ). The relationships established are often made up of conflicting evidence (e.g., a positive or significant effect in one study, but a negative or non-significant effect in another study). However, through meta-analysis, scholars are able to identify potential factors (e.g., contexts or sociodemographic information) that may have led to the conflict.

Others: Multiple correspondence analysis helps to map the field, assessing the associations between qualitative content within a matrix of variables and cases. Homogeneity Analysis by Means of Alternating Least Squares ( HOMALS ) is also considered useful in allowing researchers to map out the intellectual structure of a variety of research fields (Gonzalez-Loureiro et al. 2015 ; Gonzalez-Louriero 2021; Obradović et al. 2021 ). HOMALS can be performed in R or used along with a matrix through SPSS software. In summary, the overall objective of this analysis is to discover a low dimensional representation of the original high dimensional space (i.e., the matrix of descriptors and articles). To measure the goodness of fit, a loss function is used. This function is used minimally, and the HOMALS algorithm is applied to the least squares loss functions in SPSS. This analysis provides a proximity map, in which articles and descriptors are shown in low-dimensional spaces (typically on two axes). Keywords are paired and each couple that appears together in a large number of articles is shown to be closer on the map and vice-versa.

When conducting a literature review, software solutions allow researchers to cover a broad range of variables, from built-in functions of statistical software packages to software orientated towards meta-analyses, and from commercial to open-source solutions. Personal preference plays a huge role, but the decision as to which software will be the most useful is entirely dependent on how complex the methods and the dataset are. Of all the commercial software providers, we have found the built-in functions of (i) R and VOSviewer most useful in performing bibliometric analysis (Aria and Cuccurullo 2017 ; R Core Team 2021 ; Van Eck and Waltman 2014 ) and (ii) Stata most useful in performing meta-analytical tasks.

Many different analytical tools have been used. These include simple document counting, citation analysis, word frequency analysis, cluster analysis, co-word analysis, and cooperation analysis (Daim et al. 2006 ). Software has also been produced for bibliometric analysis, such as the Thomson Data Analyzer (TDA), which Thomson Reuters created, and CiteSpace developed by Chen ( 2013 ). VOSviewer helps us to construct and visualize bibliometric networks, which can include articles, journals, authors, countries, and institutions, among others (Van Eck and Waltman 2014 ). These can be organized based on citations, co-citations, bibliographic coupling, or co-authorship relations. In addition, VOSviewer provides text mining functions, which can be used to facilitate a better understanding of co-occurrence networks with regards to the key terms taken from a body of scientific literature (Donthu et al. 2021 ; Wong 2018 ). Other frequently used tools include for bibliometric analysis include Bibliometrix/Biblioshiny in R, CitNetExplorer, and Gephi, among others.

2.5 Contributions

Well-conducted literature reviews may make multiple contributions to the literature as standalone, independent studies.

Generally, there are three primary contributions of literature reviews as independent studies: (i) to provide an overview of current knowledge in the domain, method, or theory, (ii) to provide an evaluation of knowledge progression in the domain, method, or theory, including the establishment of key knowledge, conflicting or inconclusive findings, and emerging and underexplored areas, and (iii) to provide a proposal for potential pathways for advancing knowledge in the domain, method, or theory (Lim et al. 2022a , p. 487). Developing theory through literature reviews can take many forms, including organizing and categorizing the literature, problematizing the literature, identifying and exposing contradictions, developing analogies and metaphors, and setting out new narratives and conceptualizations (Breslin and Gatrell 2020 ). Taken collectively, these contributions offer crystalized, evidence-based insights that both ‘mine’ and ‘prospect’ the literature, highlighting extant gaps and how they can be resolved (e.g., flags paradoxes or theoretical tensions, explaining why something has not been done, what the challenges are, and how these challenges can be overcome). These contributions can be derived through successful bibliometric analysis, content analysis, critical analysis, and meta-analysis.

Additionally, the deployment of specific methods can bring in further added value. For example, a performance analysis in a bibliometric analysis can contribute to: (i) objectively assessing and reporting research productivity and impact ; (ii) ascertaining reach for coverage claims ; (iii) identifying social dominance and hidden biases ; (iv) detecting anomalies ; and (v) evaluating ( equitable ) relative performance ; whereas science mapping in bibliometric analysis can contribute to: (i) objectively discovering thematic clusters of knowledge ; (ii) clarifying nomological networks ; (iii) mapping social patterns ; (iv) tracking evolutionary nuances ; and (v) recognizing knowledge gaps (Mukherjee et al. 2022b , p. 105).

3 Conclusion

Independent literature reviews will continue to be written as a result of their necessity, importance, relevance, and urgency when it comes to advancing knowledge (Lim et al. 2022a ; Mukherjee et al. 2022b ), and this can be seen in the increasing number of reviews being published over the last several years. Literature reviews advance academic discussion. Journal publications on various topics and subject areas are becoming more frequent sites for publication. This trend will only heighten the need for literature reviews. This article offers directions and control points that address the needs of three different stakeholder groups: producers (i.e., potential authors), evaluators (i.e., journal editors and reviewers), and users (i.e., new researchers looking to learn more about a particular methodological issue, and those teaching the next generation of scholars). Future producers will derive value from this article’s teachings on the different fundamental elements and methodological nuances of literature reviews. Procedural knowledge (i.e., using control points to assist in decision-making during the manuscript preparation phase) will also be of use. Evaluators will be able to make use of the procedural and declarative knowledge evident in control points as well. As previously outlined, the need to cultivate novelty within research on business and management practices is vital. Scholars must also be supported to choose not only safe mining approaches; they should also be encouraged to attempt more challenging and risky ventures. It is important to note that abstracts often seem to offer a lot of potential, stating that authors intend to make large conceptual contributions, broadening the horizons of the field.

Our article offers important insights also for practitioners. Noteworthily, our framework can support corporate managers in decomposing and better understanding literature reviews as ad-hoc and independent studies about specific topics that matter for their organization. For instance, practitioners can understand more easily what are the emerging trends within their domain of interest and make corporate decisions in line with such trends.

This article arises from an intentional decoupling from philosophy, in favor of adopting a more pragmatic approach. This approach can assist us in clarifying the fundamental elements of literature reviews as independent studies. Five fundamental elements must be considered: types, focuses, considerations, methods, and contributions. These elements offer a useful frame for scholars starting to work on a literature review. Overview articles (guides) such as ours are thus invaluable, as they equip scholars with a solid foundational understanding of the integral elements of a literature review. Scholars can then put these teachings into practice, armed with a better understanding of the philosophy that underpins the procedures, protocols, and nomenclatures of literature reviews as independent studies.

Data availability

Our manuscript has no associate data.

Our focus here is on standalone literature reviews in contrast with literature reviews that form the theoretical foundation for a research article.

Scoping reviews, structured reviews, integrative reviews, and interpretive/narrative reviews are commonly found in review nomenclature. However, the philosophy of these review nomenclatures essentially reflects what constitutes a good SLR. That is to say, a good SLR should be well scoped, structured, integrated, and interpreted/narrated. This observation reaffirms our position and the value of moving away from review nomenclatures to gain a foundational understanding of literature reviews as independent studies.

Given that many of these considerations can be implemented simultaneously in contemporary versions of scientific databases, scholars may choose to consolidate them into a single (or a few) step(s), where appropriate, so that they can be reported more parsimoniously. For a parsimonious but transparent and replicable exemplar, see Lim ( 2022 ).

Where keywords are present (e.g., author keywords or keywords derived from machine learning [e.g., natural language processing]), it is assumed that each keyword represents a specific meaning (e.g., topic [concept, context], method), and that a collection of keywords grouped under the same cluster represents a specific theme.

Aguinis H, Jensen SH, Kraus S (2022) Policy implications of organizational behavior and human resource management research. Acad Manage Perspect 36(3):1–22

Article   Google Scholar  

Ammirato S, Felicetti AM, Rogano D, Linzalone R, Corvello V (2022) Digitalising the systematic literature review process: The My SLR platform. Knowl Manage Res Pract. doi: https://doi.org/10.1080/14778238.2022.2041375

Aria M, Cuccurullo C (2017) bibliometrix: An R-tool for comprehensive science mapping analysis. J Informetrics 11(4):959–975

Baker WE, Mukherjee D, Perin MG (2022) Learning orientation and competitive advantage: A critical synthesis and future directions. J Bus Res 144:863–873

Boell SK, Cecez-Kecmanovic D (2015) On being ‘systematic’ in literature reviews. J Inform Technol 30:161–173

Borrego M, Foster MJ, Froyd JE (2014) Systematic literature reviews in engineering education and other developing interdisciplinary fields. J Eng Educ 103(1):45–76

Breslin D, Gatrell C (2020) Theorizing through literature reviews: The miner-prospector continuum. Organizational Res Methods. https://doi.org/10.1177/1094428120943288 (in press)

Cajal B, Jiménez R, Gervilla E, Montaño JJ (2020) Doing a systematic review in health sciences. Clínica y Salud 31(2):77–83

Chen C (2013) Mapping scientific frontiers: The quest for knowledge visualization. Springer Science & Business Media

Creevey D, Coughlan J, O’Connor C (2022) Social media and luxury: A systematic literature review. Int J Manage Reviews 24(1):99–129

Dabić M, González-Loureiro M, Harvey M (2015) Evolving research on expatriates: what is ‘known’after four decades (1970–2012). Int J Hum Resource Manage 26(3):316–337

Dabić M, Vlačić B, Kiessling T, Caputo A, Pellegrini M(2021) Serial entrepreneurs: A review of literature and guidance for future research.Journal of Small Business Management,1–36

Daim TU, Rueda G, Martin H, Gerdsri P (2006) Forecasting emerging technologies: Use of bibliometrics and patent analysis. Technol Forecast Soc Chang 73(8):981–1012

Deng L, Yang M, Marcoulides KM (2018) Structural equation modeling with many variables: A systematic review of issues and developments. Front Psychol 9:580

Donthu N, Kumar S, Pattnaik D (2020) Forty-five years of Journal of Business Research: A bibliometric analysis. J Bus Res 109:1–14

Donthu N, Kumar S, Mukherjee D, Pandey N, Lim WM (2021) How to conduct a bibliometric analysis: An overview and guidelines. J Bus Res 133:285–296

Duan W, Jiang G (2008) A review of the theory of planned behavior. Adv Psychol Sci 16(2):315–320

Google Scholar  

Durach CF, Kembro J, Wieland A (2017) A new paradigm for systematic literature reviews in supply chain management. J Supply Chain Manage 53(4):67–85

Fan D, Breslin D, Callahan JL, Szatt-White M (2022) Advancing literature review methodology through rigour, generativity, scope and transparency. Int J Manage Reviews 24(2):171–180

Ferreira MP, Reis NR, Miranda R (2015) Thirty years of entrepreneurship research published in top journals: Analysis of citations, co-citations and themes. J Global Entrepreneurship Res 5(1):1–22

Ghauri P, Strange R, Cooke FL (2021) Research on international business: The new realities. Int Bus Rev 30(2):101794

Gioia DA, Corley KG, Hamilton AL (2012) Seeking qualitative rigor in inductive research: Notes on the gioia methodology. Organizational Res Methods 16(1):15–31

Gonzalez-Loureiro M, Dabić M, Kiessling T (2015) Supply chain management as the key to a firm’s strategy in the global marketplace: Trends and research agenda. Int J Phys Distribution Logistics Manage 45(1/2):159–181. https://doi.org/10.1108/IJPDLM-05-2013-0124

Grewal D, Puccinelli N, Monroe KB (2018) Meta-analysis: Integrating accumulated knowledge. J Acad Mark Sci 46(1):9–30

Hansen C, Steinmetz H, Block J(2021) How to conduct a meta-analysis in eight steps: a practical guide.Management Review Quarterly,1–19

Korherr P, Kanbach DK (2021) Human-related capabilities in big data analytics: A taxonomy of human factors with impact on firm performance. RMS. https://doi.org/10.1007/s11846-021-00506-4 (in press)

Kraus S, Breier M, Dasí-Rodríguez S (2020) The art of crafting a systematic literature review in entrepreneurship research. Int Entrepreneurship Manage J 16(3):1023–1042

Kraus S, Durst S, Ferreira J, Veiga P, Kailer N, Weinmann A (2022) Digital transformation in business and management research: An overview of the current status quo. Int J Inf Manag 63:102466

Kraus S, Jones P, Kailer N, Weinmann A, Chaparro-Banegas N, Roig-Tierno N (2021) Digital transformation: An overview of the current state of the art of research. Sage Open 11(3):1–15

Kraus S, Mahto RV, Walsh ST (2021) The importance of literature reviews in small business and entrepreneurship research. J Small Bus Manage. https://doi.org/10.1080/00472778.2021.1955128 (in press)

Kumar S, Sahoo S, Lim WM, Dana LP (2022a) Religion as a social shaping force in entrepreneurship and business: Insights from a technology-empowered systematic literature review. Technol Forecast Soc Chang 175:121393

Kumar S, Sahoo S, Lim WM, Kraus S, Bamel U (2022b) Fuzzy-set qualitative comparative analysis (fsQCA) in business and management research: A contemporary overview. Technol Forecast Soc Chang 178:121599

Kumar S, Sharma D, Rao S, Lim WM, Mangla SK (2022c) Past, present, and future of sustainable finance: Insights from big data analytics through machine learning of scholarly research. Ann Oper Res. https://doi.org/10.1007/s10479-021-04410-8 (in press)

Laher S, Hassem T (2020) Doing systematic reviews in psychology. South Afr J Psychol 50(4):450–468

Leemann N, Kanbach DK (2022) Toward a taxonomy of dynamic capabilities – a systematic literature review. Manage Res Rev 45(4):486–501

Lahiri S, Mukherjee D, Peng MW (2020) Behind the internationalization of family SMEs: A strategy tripod synthesis. Glob Strategy J 10(4):813–838

Lim WM (2018) Demystifying neuromarketing. J Bus Res 91:205–220

Lim WM (2020) The sharing economy: A marketing perspective. Australasian Mark J 28(3):4–13

Lim WM (2022) Ushering a new era of Global Business and Organizational Excellence: Taking a leaf out of recent trends in the new normal. Global Bus Organizational Excellence 41(5):5–13

Lim WM, Rasul T (2022) Customer engagement and social media: Revisiting the past to inform the future. J Bus Res 148:325–342

Lim WM, Weissmann MA (2021) Toward a theory of behavioral control. J Strategic Mark. https://doi.org/10.1080/0965254X.2021.1890190 (in press)

Lim WM, Kumar S, Ali F (2022a) Advancing knowledge through literature reviews: ‘What’, ‘why’, and ‘how to contribute’. Serv Ind J 42(7–8):481–513

Lim WM, Rasul T, Kumar S, Ala M (2022b) Past, present, and future of customer engagement. J Bus Res 140:439–458

Lim WM, Yap SF, Makkar M (2021) Home sharing in marketing and tourism at a tipping point: What do we know, how do we know, and where should we be heading? J Bus Res 122:534–566

López-Duarte C, González-Loureiro M, Vidal-Suárez MM, González-Díaz B (2016) International strategic alliances and national culture: Mapping the field and developing a research agenda. J World Bus 51(4):511–524

Mas-Tur A, Kraus S, Brandtner M, Ewert R, Kürsten W (2020) Advances in management research: A bibliometric overview of the Review of Managerial Science. RMS 14(5):933–958

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Reviews 4(1):1–9

Mukherjee D, Kumar S, Donthu N, Pandey N (2021) Research published in Management International Review from 2006 to 2020: A bibliometric analysis and future directions. Manage Int Rev 61:599–642

Mukherjee D, Kumar S, Mukherjee D, Goyal K (2022a) Mapping five decades of international business and management research on India: A bibliometric analysis and future directions. J Bus Res 145:864–891

Mukherjee D, Lim WM, Kumar S, Donthu N (2022b) Guidelines for advancing theory and practice through bibliometric research. J Bus Res 148:101–115

Obradović T, Vlačić B, Dabić M (2021) Open innovation in the manufacturing industry: A review and research agenda. Technovation 102:102221

Ojong N, Simba A, Dana LP (2021) Female entrepreneurship in Africa: A review, trends, and future research directions. J Bus Res 132:233–248

Palmatier RW, Houston MB, Hulland J (2018) Review articles: Purpose, process, and structure. J Acad Mark Sci 46(1):1–5

Post C, Sarala R, Gatrell C, Prescott JE (2020) Advancing theory with review articles. J Manage Stud 57(2):351–376

Pranckutė R (2021) Web of Science (WoS) and Scopus: The titans of bibliographic information in today’s academic world. Publications 9(1):12

R Core Team (2021) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/ Accessed 20th July 2022

Rovelli P, Ferasso M, De Massis A, Kraus S(2021) Thirty years of research in family business journals: Status quo and future directions.Journal of Family Business Strategy,100422

Salmony FU, Kanbach DK (2022) Personality trait differences across types of entrepreneurs: a systematic literature review. RMS 16:713–749

Snyder H (2019) Literature review as a research methodology: An overview and guidelines. J Bus Res 104:333–339

Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag 14(3):207–222

Ulker-Demirel E, Ciftci G (2020) A systematic literature review of the theory of planned behavior in tourism, leisure and hospitality management research. J Hospitality Tourism Manage 43:209–219

Van Eck NJ, Waltma L (2014) CitNetExplorer: A new software tool for analyzing and visualizing citation networks. J Informetrics 8(4):802–823

Vlačić B, Corbo L, Silva e, Dabić M (2021) The evolving role of artificial intelligence in marketing: A review and research agenda. J Bus Res 128:187–203

Wong D (2018) VOSviewer. Tech Serv Q 35(2):219–220. https://doi.org/10.1080/07317131.2018.1425352

Download references

Open access funding provided by Libera Università di Bolzano within the CRUI-CARE Agreement.

Author information

Authors and affiliations.

Faculty of Economics & Management, Free University of Bozen-Bolzano, Bolzano, Italy

Sascha Kraus

Department of Business Management, University of Johannesburg, Johannesburg, South Africa

School of Business, Lappeenranta University of Technology, Lappeenranta, Finland

Matthias Breier

Sunway University Business School, Sunway University, Sunway City, Malaysia

Weng Marc Lim

Faculty of Economics and Business, University of Zagreb, Zagreb, Croatia

Marina Dabić

School of Economics and Business, University of Ljubljana, Ljubljana, Slovenia

Department of Management Studies, Malaviya National Institute of Technology Jaipur, Jaipur, India

Satish Kumar

Faculty of Business, Design and Arts, Swinburne University of Technology, Kuching, Malaysia

Weng Marc Lim & Satish Kumar

Chair of Strategic Management and Digital Entrepreneurship, HHL Leipzig Graduate School of Management, Leipzig, Germany

Dominik Kanbach

School of Business, Woxsen University, Hyderabad, India

College of Business, The University of Akron, Akron, USA

Debmalya Mukherjee

Department of Engineering, University of Messina, Messina, Italy

Vincenzo Corvello

Department of Finance, Santiago de Compostela University, Santiago de Compostela, Spain

Juan Piñeiro-Chousa

Rowan University, Rohrer College of Business, Glassboro, NJ, USA

Eric Liguori

School of Engineering Design, Universidad Politécnica de Valencia, Valencia, Spain

Daniel Palacios-Marqués

Department of Management and Quantitative Studies, Parthenope University, Naples, Italy

Francesco Schiavone

Paris School of Business, Paris, France

Department of Management, University of Turin, Turin, Italy

Alberto Ferraris

Department of Management and Economics & NECE Research Unit in Business Sciences, University of Beira Interior, Covilha, Portugal

Cristina Fernandes & João J. Ferreira

Centre for Corporate Entrepreneurship and Innovation, Loughborough University, Loughborough, UK

Cristina Fernandes

Laboratory for International and Regional Economics, Graduate School of Economics and Management, Ural Federal University, Yekaterinburg, Russia

School of Business, Law and Entrepreneurship, Swinburne University of Technology, Hawthorn, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sascha Kraus .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Kraus, S., Breier, M., Lim, W.M. et al. Literature reviews as independent studies: guidelines for academic practice. Rev Manag Sci 16 , 2577–2595 (2022). https://doi.org/10.1007/s11846-022-00588-8

Download citation

Received : 15 August 2022

Accepted : 07 September 2022

Published : 14 October 2022

Issue Date : November 2022

DOI : https://doi.org/10.1007/s11846-022-00588-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Literature reviews
• Bibliometrics
• Meta Analysis
• Contributions

JEL classification

• Find a journal
• Publish with us
• Track your research

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• What Is Peer Review? | Types & Examples

What Is Peer Review? | Types & Examples

Published on December 17, 2021 by Tegan George . Revised on June 22, 2023.

Peer review, sometimes referred to as refereeing , is the process of evaluating submissions to an academic journal. Using strict criteria, a panel of reviewers in the same subject area decides whether to accept each submission for publication.

Peer-reviewed articles are considered a highly credible source due to the stringent process they go through before publication.

There are various types of peer review. The main difference between them is to what extent the authors, reviewers, and editors know each other’s identities. The most common types are:

• Single-blind review
• Double-blind review
• Triple-blind review

Collaborative review

Open review.

Relatedly, peer assessment is a process where your peers provide you with feedback on something you’ve written, based on a set of criteria or benchmarks from an instructor. They then give constructive feedback, compliments, or guidance to help you improve your draft.

Table of contents

What is the purpose of peer review, types of peer review, the peer review process, providing feedback to your peers, peer review example, advantages of peer review, criticisms of peer review, other interesting articles, frequently asked questions about peer reviews.

Many academic fields use peer review, largely to determine whether a manuscript is suitable for publication. Peer review enhances the credibility of the manuscript. For this reason, academic journals are among the most credible sources you can refer to.

However, peer review is also common in non-academic settings. The United Nations, the European Union, and many individual nations use peer review to evaluate grant applications. It is also widely used in medical and health-related fields as a teaching or quality-of-care measure.

Peer assessment is often used in the classroom as a pedagogical tool. Both receiving feedback and providing it are thought to enhance the learning process, helping students think critically and collaboratively.

Prevent plagiarism. Run a free check.

Depending on the journal, there are several types of peer review.

Single-blind peer review

The most common type of peer review is single-blind (or single anonymized) review . Here, the names of the reviewers are not known by the author.

While this gives the reviewers the ability to give feedback without the possibility of interference from the author, there has been substantial criticism of this method in the last few years. Many argue that single-blind reviewing can lead to poaching or intellectual theft or that anonymized comments cause reviewers to be too harsh.

Double-blind peer review

In double-blind (or double anonymized) review , both the author and the reviewers are anonymous.

Arguments for double-blind review highlight that this mitigates any risk of prejudice on the side of the reviewer, while protecting the nature of the process. In theory, it also leads to manuscripts being published on merit rather than on the reputation of the author.

Triple-blind peer review

While triple-blind (or triple anonymized) review —where the identities of the author, reviewers, and editors are all anonymized—does exist, it is difficult to carry out in practice.

Proponents of adopting triple-blind review for journal submissions argue that it minimizes potential conflicts of interest and biases. However, ensuring anonymity is logistically challenging, and current editing software is not always able to fully anonymize everyone involved in the process.

In collaborative review , authors and reviewers interact with each other directly throughout the process. However, the identity of the reviewer is not known to the author. This gives all parties the opportunity to resolve any inconsistencies or contradictions in real time, and provides them a rich forum for discussion. It can mitigate the need for multiple rounds of editing and minimize back-and-forth.

Collaborative review can be time- and resource-intensive for the journal, however. For these collaborations to occur, there has to be a set system in place, often a technological platform, with staff monitoring and fixing any bugs or glitches.

Lastly, in open review , all parties know each other’s identities throughout the process. Often, open review can also include feedback from a larger audience, such as an online forum, or reviewer feedback included as part of the final published product.

While many argue that greater transparency prevents plagiarism or unnecessary harshness, there is also concern about the quality of future scholarship if reviewers feel they have to censor their comments.

In general, the peer review process includes the following steps:

• First, the author submits the manuscript to the editor.
• Reject the manuscript and send it back to the author, or
• Send it onward to the selected peer reviewer(s)
• Next, the peer review process occurs. The reviewer provides feedback, addressing any major or minor issues with the manuscript, and gives their advice regarding what edits should be made.
• Lastly, the edited manuscript is sent back to the author. They input the edits and resubmit it to the editor for publication.

In an effort to be transparent, many journals are now disclosing who reviewed each article in the published product. There are also increasing opportunities for collaboration and feedback, with some journals allowing open communication between reviewers and authors.

It can seem daunting at first to conduct a peer review or peer assessment. If you’re not sure where to start, there are several best practices you can use.

Summarize the argument in your own words

Summarizing the main argument helps the author see how their argument is interpreted by readers, and gives you a jumping-off point for providing feedback. If you’re having trouble doing this, it’s a sign that the argument needs to be clearer, more concise, or worded differently.

If the author sees that you’ve interpreted their argument differently than they intended, they have an opportunity to address any misunderstandings when they get the manuscript back.

Separate your feedback into major and minor issues

It can be challenging to keep feedback organized. One strategy is to start out with any major issues and then flow into the more minor points. It’s often helpful to keep your feedback in a numbered list, so the author has concrete points to refer back to.

Major issues typically consist of any problems with the style, flow, or key points of the manuscript. Minor issues include spelling errors, citation errors, or other smaller, easy-to-apply feedback.

Tip: Try not to focus too much on the minor issues. If the manuscript has a lot of typos, consider making a note that the author should address spelling and grammar issues, rather than going through and fixing each one.

The best feedback you can provide is anything that helps them strengthen their argument or resolve major stylistic issues.

Give the type of feedback that you would like to receive

No one likes being criticized, and it can be difficult to give honest feedback without sounding overly harsh or critical. One strategy you can use here is the “compliment sandwich,” where you “sandwich” your constructive criticism between two compliments.

Be sure you are giving concrete, actionable feedback that will help the author submit a successful final draft. While you shouldn’t tell them exactly what they should do, your feedback should help them resolve any issues they may have overlooked.

As a rule of thumb, your feedback should be:

• Easy to understand
• Constructive

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Below is a brief annotated research example. You can view examples of peer feedback by hovering over the highlighted sections.

Influence of phone use on sleep

Studies show that teens from the US are getting less sleep than they were a decade ago (Johnson, 2019) . On average, teens only slept for 6 hours a night in 2021, compared to 8 hours a night in 2011. Johnson mentions several potential causes, such as increased anxiety, changed diets, and increased phone use.

The current study focuses on the effect phone use before bedtime has on the number of hours of sleep teens are getting.

For this study, a sample of 300 teens was recruited using social media, such as Facebook, Instagram, and Snapchat. The first week, all teens were allowed to use their phone the way they normally would, in order to obtain a baseline.

The sample was then divided into 3 groups:

• Group 1 was not allowed to use their phone before bedtime.
• Group 2 used their phone for 1 hour before bedtime.
• Group 3 used their phone for 3 hours before bedtime.

All participants were asked to go to sleep around 10 p.m. to control for variation in bedtime . In the morning, their Fitbit showed the number of hours they’d slept. They kept track of these numbers themselves for 1 week.

Two independent t tests were used in order to compare Group 1 and Group 2, and Group 1 and Group 3. The first t test showed no significant difference ( p > .05) between the number of hours for Group 1 ( M = 7.8, SD = 0.6) and Group 2 ( M = 7.0, SD = 0.8). The second t test showed a significant difference ( p < .01) between the average difference for Group 1 ( M = 7.8, SD = 0.6) and Group 3 ( M = 6.1, SD = 1.5).

This shows that teens sleep fewer hours a night if they use their phone for over an hour before bedtime, compared to teens who use their phone for 0 to 1 hours.

Peer review is an established and hallowed process in academia, dating back hundreds of years. It provides various fields of study with metrics, expectations, and guidance to ensure published work is consistent with predetermined standards.

• Protects the quality of published research

Peer review can stop obviously problematic, falsified, or otherwise untrustworthy research from being published. Any content that raises red flags for reviewers can be closely examined in the review stage, preventing plagiarized or duplicated research from being published.

• Gives you access to feedback from experts in your field

Peer review represents an excellent opportunity to get feedback from renowned experts in your field and to improve your writing through their feedback and guidance. Experts with knowledge about your subject matter can give you feedback on both style and content, and they may also suggest avenues for further research that you hadn’t yet considered.

• Helps you identify any weaknesses in your argument

Peer review acts as a first defense, helping you ensure your argument is clear and that there are no gaps, vague terms, or unanswered questions for readers who weren’t involved in the research process. This way, you’ll end up with a more robust, more cohesive article.

While peer review is a widely accepted metric for credibility, it’s not without its drawbacks.

• Reviewer bias

The more transparent double-blind system is not yet very common, which can lead to bias in reviewing. A common criticism is that an excellent paper by a new researcher may be declined, while an objectively lower-quality submission by an established researcher would be accepted.

• Delays in publication

The thoroughness of the peer review process can lead to significant delays in publishing time. Research that was current at the time of submission may not be as current by the time it’s published. There is also high risk of publication bias , where journals are more likely to publish studies with positive findings than studies with negative findings.

• Risk of human error

By its very nature, peer review carries a risk of human error. In particular, falsification often cannot be detected, given that reviewers would have to replicate entire experiments to ensure the validity of results.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Measures of central tendency
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Thematic analysis
• Discourse analysis
• Cohort study
• Ethnography

Research bias

• Implicit bias
• Cognitive bias
• Conformity bias
• Hawthorne effect
• Availability heuristic
• Attrition bias
• Social desirability bias

Peer review is a process of evaluating submissions to an academic journal. Utilizing rigorous criteria, a panel of reviewers in the same subject area decide whether to accept each submission for publication. For this reason, academic journals are often considered among the most credible sources you can use in a research project– provided that the journal itself is trustworthy and well-regarded.

In general, the peer review process follows the following steps: 

• Reject the manuscript and send it back to author, or 
• Send it onward to the selected peer reviewer(s) 
• Next, the peer review process occurs. The reviewer provides feedback, addressing any major or minor issues with the manuscript, and gives their advice regarding what edits should be made. 
• Lastly, the edited manuscript is sent back to the author. They input the edits, and resubmit it to the editor for publication.

Peer review can stop obviously problematic, falsified, or otherwise untrustworthy research from being published. It also represents an excellent opportunity to get feedback from renowned experts in your field. It acts as a first defense, helping you ensure your argument is clear and that there are no gaps, vague terms, or unanswered questions for readers who weren’t involved in the research process.

Peer-reviewed articles are considered a highly credible source due to this stringent process they go through before publication.

Many academic fields use peer review , largely to determine whether a manuscript is suitable for publication. Peer review enhances the credibility of the published manuscript.

However, peer review is also common in non-academic settings. The United Nations, the European Union, and many individual nations use peer review to evaluate grant applications. It is also widely used in medical and health-related fields as a teaching or quality-of-care measure. 

A credible source should pass the CRAAP test  and follow these guidelines:

• The information should be up to date and current.
• The author and publication should be a trusted authority on the subject you are researching.
• The sources the author cited should be easy to find, clear, and unbiased.
• For a web source, the URL and layout should signify that it is trustworthy.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). What Is Peer Review? | Types & Examples. Scribbr. Retrieved July 26, 2024, from https://www.scribbr.com/methodology/peer-review/

Is this article helpful?

Tegan George

Other students also liked, what are credible sources & how to spot them | examples, ethical considerations in research | types & examples, applying the craap test & evaluating sources, what is your plagiarism score.

• Interlibrary Loan and Scan & Deliver
• Course Reserves
• Purchase Request
• Collection Development & Maintenance
• Current Negotiations
• Ask a Librarian
• Instructor Support
• Library How-To
• Research Guides
• Research Support
• Study Rooms
• Research Rooms
• Partner Spaces
• Loanable Equipment
• Print, Scan, Copy
• 3D Printers
• Poster Printing
• OSULP Leadership
• Strategic Plan

Scholarly Articles: How can I tell?

• Journal Information

Literature Review

• Author and affiliation
• Introduction
• Specialized Vocabulary
• Methodology
• Research sponsors
• Peer-review

The literature review section of an article is a summary or analysis of all the research the author read before doing his/her own research. This section may be part of the introduction or in a section called Background. It provides the background on who has done related research, what that research has or has not uncovered and how the current research contributes to the conversation on the topic. When you read the lit review ask:

• Does the review of the literature logically lead up to the research questions?
• Do the authors review articles relevant to their research study?
• Do the authors show where there are gaps in the literature?

The lit review is also a good place to find other sources you may want to read on this topic to help you get the bigger picture.

• << Previous: Journal Information
• Next: Author and affiliation >>
• Last Updated: Apr 15, 2024 3:26 PM
• URL: https://guides.library.oregonstate.edu/ScholarlyArticle

Contact Info

121 The Valley Library Corvallis OR 97331–4501

Phone: 541-737-3331

Services for Persons with Disabilities

In the Valley Library

• Oregon State University Press
• Special Collections and Archives Research Center
• Undergrad Research & Writing Studio
• Graduate Student Commons
• Tutoring Services
• Northwest Art Collection

Digital Projects

• Oregon Explorer
• Oregon Digital
• ScholarsArchive@OSU
• Digital Publishing Initiatives
• Atlas of the Pacific Northwest
• Marilyn Potts Guin Library  
• Cascades Campus Library
• McDowell Library of Vet Medicine

University Libraries      University of Nevada, Reno

• Skill Guides
• Subject Guides

Literature Reviews

• Searching for Literature
• Organizing Literature and Taking Notes
• Writing and Editing the Paper
• Help and Resources

Book an Appointment

What Is a Literature Review?

A literature review surveys and synthesizes the scholarly research literature related to a particular topic. Literature reviews both explain research findings and analyze the quality of the research in order to arrive at new insights.

Literature reviews may describe not only the key research related to a topic of inquiry but also seminal sources, influential scholars, key theories or hypotheses, common methodologies used, typical questions asked, or common patterns of inquiry.

There are different types of literature reviews.  A narrative literature review summarizes and synthesizes the findings of numerous research articles, but the purpose and scope of narrative literature reviews vary widely. The term "literature review" is most commonly used to refer to narrative literature reviews, and these are the types of works that are described in this guide. 

Some types of literature reviews that use prescribed methods for identifying and evaluating evidence-based literature related to specific questions are known as systematic reviews or meta-analyses . Systematic reviews or meta-analyses are typically conducted by at least two scholars working in collaboration as prescribed by certain guidelines, but narrative literature reviews may be conducted by authors working alone.

Purpose of a Literature Review

Literature reviews serve an important function in developing the scholarly record. Because of the vast amount of scholarly literature that exists, it can be difficult for readers to keep up with the latest developments related to a topic, or to discern which ideas, themes, authors, or methods are worthy of more attention. Literature reviews help readers to understand and make sense of a large body of scholarship.

Literature reviews also play an important function in assessing the quality of the evidence base in relation to a particular topic. Literature reviews contain assessments of the evidence in support of particular interventions, policies, programs, or treatments.

The literature that is reviewed may include a variety of types of research, including empirical research, theoretical works, and reports of practical application. The scholarly works that are considered for inclusion in a literature review may appear in a variety of publication types, including scholarly journals, books, conference proceedings, reports, and others. 

Steps in the Process

Follow these steps to conduct your literature review:

• Select a topic and prepare for searching.  Formulate a research question and establish inclusion and exclusion criteria for your search.
• Search for and organize the research. Use tools like the library website, library-subscription databases, Google Scholar, and others to locate research on your topic.
• Organize your research, read and evaluate it, and take notes. Use organizational and note-taking strategies to read sources and prepare for writing. 
• Write and edit the paper. Synthesize information from sources to arrive at new insights.

Literature Reviews: An Overview for Graduate Students

View the video below for an overview of the process of writing literature review papers.

Video:  Literature Reviews: An Overview for Graduate Students  by  libncsu

• Next: Searching for Literature >>

The History of Peer Review Is More Interesting Than You Think

The term “peer review” was coined in the 1970s, but the referee principle is usually assumed to be as old as the scientific enterprise itself. (It isn’t.)

Peer review has become a cornerstone of academic publishing, a fundamental part of scholarship itself. With peer review, independent, third-party experts in the relevant field(s) assess manuscripts submitted to journals. The idea is that these expert peers referee the process, especially when it comes to technical matters that may be beyond the knowledge of editors.

“In all fields of academia, reputations and careers are now expected to be built on peer-reviewed publication; concerns with its efficacy and appropriateness thus seem to strike at the heart of scholarship,” write historians Noah Moxham and Aileen Fyfe .

The peer review system, continue Moxham and Fyfe, is “crucial to building the reputation both of individual scientists and of the scientific enterprise at large” because the process

is believed to certify the quality and reliability of research findings. It promises supposedly impartial evaluation of research, through close scrutiny by specialists, and is widely used by journal editors, grant-making bodies, and government.

As with any human enterprise, peer review is far from foolproof . Errors and downright frauds have made it through the process. In addition, as Moxham and Fyfe note, there can be “inappropriate bias due to the social dynamics of the process.” (Some peer review types may introduce less bias than others.)

The term “peer review” was coined in the early 1970s, but the referee principle is usually assumed to be about as old as the scientific enterprise itself, dating to the Royal Society of London’s Philosophical Transactions , which began publication in 1665.

Moxham and Fyfe complicate this history, using the Royal Society’s “rich archives” to trace the evolution of editorial practices at one of the earliest scientific societies.

Initially, the publication of Philosophical Transactions was a private venture managed by the Society’s secretaries. Secretary Henry Oldenburg, the first editor, ran it from 1665 to 1677, without, write Moxham and Fyfe, any “clear set of standards.”

Research sponsored by the Royal Society itself was published separately from the Transactions . In fact, the royally chartered Society had the power to license publication of books and periodicals (like the Transactions ) as “part of a wider mechanism of state censorship intended to ensure the proscription of politically seditious or religious heterodox material.” But as time passed, there wasn’t really much Society oversight over the publication at all.

The situation came to a crisis in the early 1750s, when an unsuccessful candidate for a Society fellowship raised a ruckus, conflating the separate administrations of the Society and the now rather stodgy Transactions. The bad press compelled the Society to take over financial and editorial control—by committee—of the Transactions in 1752. The editorial committee could refer submissions to fellows with particular expertise—but papers were already being vetted since they needed to be referred by fellows in the first place.

Formalization of the use of expert referees would be institutionalized by 1832. A “written report of fitness” of submissions by one or more fellows was to be made before acceptance. This followed similar procedures already introduced abroad, particularly at the Académie des sciences in Paris.

All of this, Moxham and Fyfe argue, was more about institution-building (and fortification) than what we know as peer reviewing today.

“Refereeing and associated editorial practices” were intended to “disarm specific attacks upon the eighteenth-century Society; sometimes, to protect the Society’s finances; and, by the later nineteenth century, to award prestige to members of the nascent profession of natural scientists.”

Weekly Newsletter

Get your fix of JSTOR Daily’s best stories in your inbox each Thursday.

Privacy Policy   Contact Us You may unsubscribe at any time by clicking on the provided link on any marketing message.

From 1752 to 1957, the front of every Transactions included an “ Advertisement ” noting that the Society could not pretend to “answer for the certainty of the facts, or propriety of the reasonings” of the papers contained within; all that “must still rest on the credit or judgement of their respective authors.”

The twentieth century saw a plethora of independent scientific journals and an exponential increase in scientific papers. “Professional, international scientific research” burst the bounds of the old learned societies with their gentlemanly ways. In 1973, the journal Nature (founded in 1869) made refereeing standard practice, to “raise the journal above accusations of cronyism and elitism.” Since then, peer review, as it came to be called in preference to refereeing, has become universal. At least in avowed “peer-reviewed journals.”

Support JSTOR Daily! Join our membership program on Patreon today.

JSTOR is a digital library for scholars, researchers, and students. JSTOR Daily readers can access the original research behind our articles for free on JSTOR.

Get Our Newsletter

More stories.

• From Gamification to Game-Based Learning

Brunei: A Tale of Soil and Oil

Why Architects Need Philosophy to Guide the AI Design Revolution

Inside China’s Psychoboom

Recent posts.

• The Magical Furniture of David Roentgen
• Rosalind Franklin’s Methods of Discovery
• Out with a Whimper
• The Enduring Drive-In Theater

Support JSTOR Daily

Sign up for our weekly newsletter.

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Measurement of changes to the menstrual cycle: A transdisciplinary systematic review evaluating measure quality and utility for clinical trials

Roles Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Global Health and Population, FHI 360, Washington, District of Columbia, United States of America

Roles Formal analysis, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing

Affiliations Global Health and Population, FHI 360, Durham, North Carolina, United States of America, Department of Maternal and Child Health, University of North Carolina Gillings School of Global Public Health, Chapel Hill, North Carolina, United States of America

Roles Conceptualization, Formal analysis, Investigation, Project administration, Writing – original draft, Writing – review & editing

Affiliation Global Health and Population, FHI 360, Durham, North Carolina, United States of America

Roles Formal analysis, Investigation, Resources, Writing – review & editing

Current address: Independent Consultant, Washington, District of Columbia, United States of America

Affiliation Research, Technology and Utilization Division, United States Agency for International Development and the Public Health Institute, Office of Population and Reproductive Health, Bureau for Global Health, Washington, District of Columbia, United States of America

Roles Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

Current address: Guttmacher Institute, New York, New York, United States of America

• Amelia C. L. Mackenzie, 
• Stephanie Chung, 
• Emily Hoppes, 
• Alexandria K Mickler, 
• Alice F. Cartwright

• Published: July 25, 2024
• https://doi.org/10.1371/journal.pone.0306491
• Peer Review
• Reader Comments

Despite the importance of menstruation and the menstrual cycle to health, human rights, and sociocultural and economic wellbeing, the study of menstrual health suffers from a lack of funding, and research remains fractured across many disciplines. We sought to systematically review validated approaches to measure four aspects of changes to the menstrual cycle—bleeding, blood, pain, and perceptions—caused by any source and used within any field. We then evaluated the measure quality and utility for clinical trials of the identified instruments. We searched MEDLINE, Embase, and four instrument databases and included peer-reviewed articles published between 2006 and 2023 that reported on the development or validation of instruments assessing menstrual changes using quantitative or mixed-methods methodology. From a total of 8,490 articles, 8,316 were excluded, yielding 174 articles reporting on 94 instruments. Almost half of articles were from the United States or United Kingdom and over half of instruments were only in English, Spanish, French, or Portuguese. Most instruments measured bleeding parameters, uterine pain, or perceptions, but few assessed characteristics of blood. Nearly 60% of instruments were developed for populations with menstrual or gynecologic disorders or symptoms. Most instruments had fair or good measure quality or clinical trial utility; however, most instruments lacked evidence on responsiveness, question sensitivity and/or transferability, and only three instruments had good scores of both quality and utility. Although we took a novel, transdisciplinary approach, our systematic review found important gaps in the literature and instrument landscape, pointing towards a need to examine the menstrual cycle in a more comprehensive, inclusive, and standardized way. Our findings can inform the development of new or modified instruments, which—if used across the many fields that study menstrual health and within clinical trials—can contribute to a more systemic and holistic understanding of menstruation and the menstrual cycle.

Citation: Mackenzie ACL, Chung S, Hoppes E, Mickler AK, Cartwright AF (2024) Measurement of changes to the menstrual cycle: A transdisciplinary systematic review evaluating measure quality and utility for clinical trials. PLoS ONE 19(7): e0306491. https://doi.org/10.1371/journal.pone.0306491

Editor: Alison Parker, Cranfield University, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND

Received: January 23, 2024; Accepted: June 18, 2024; Published: July 25, 2024

This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: This review is made possible by the generous support of the American people through the U.S. Agency for International Development (USAID), provided to FHI 360 through cooperative agreement 7200AA20CA00016. The contents are the responsibility of FHI 360 and do not necessarily reflect the views of USAID or the United States Government.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Menstrual health across disciplines.

Menstruation and the wider menstrual cycle play a notable role in the health, human rights, and sociocultural and economic wellbeing of people who menstruate [ 1 ]. In addition, although its significance should not be utilitarianly reduced to only reproductive function, continuity of the human species would not occur without the menstrual cycle. Despite its importance, the study of menstruation and the menstrual cycle continues to suffer from a historical lack of funding and research across disciplines, including within the biological, clinical, public health, and social sciences. Within biomedical research, for example, a publication reporting on a recent technical meeting on menstruation convened by the United States National Institutes of Health (NIH) decried a “lack of understanding of basic uterine and menstrual physiology” among researchers [ 2 ]. Indeed, many foundational, field-defining works have only recently emerged in the past five to ten years following increased attention to menstrual health, which the Global Menstrual Collective defined in 2021 as “a state of complete physical, mental, and social well-being and not merely the absence of disease or infirmity, in relation to the menstrual cycle” [ 3 ]. The contemporary growth of the menstrual health field is—at least partly—due to grassroots menstrual activism, which resulted in 2015 being labeled as “the year of the period” in the lay press [ 4 ]. Other examples of recent fundamental work within menstrual health across disciplines include recommendations for the menstrual cycle to be considered a vital sign and the advent of the field of critical menstruation studies [ 5 , 6 ]. Despite these recent efforts, insufficient research on menstrual health persists. In addition, the study of menstrual health remains fractured across many fields and disciplines, many of which are siloed despite adjacent or even overlapping subject matters (e.g., menstrual health and hygiene within wider sexual and reproductive health; or gynecology, endocrinology, and many other specialties within medicine) [ 7 , 8 ]. As a result, we still lack a complete, systemic, and holistic understanding of menstruation and the wider menstrual cycle.

The type of interdisciplinary, comprehensive global efforts needed to address such large gaps in menstrual health research can greatly benefit from standardization—of terminology, of measurement, of analysis, and of outcomes or indicators. The widest global effort at standardization to date has taken place within medicine; the International Federation of Gynecology and Obstetrics (FIGO) established clinical standards of normal and abnormal uterine bleeding occurring outside of pregnancy via a consensus-building process over a series of years [ 9 – 12 ]. These FIGO standards dictate four parameters for menstrual bleeding: the frequency, duration, volume, and regularity of bleeding. FIGO defines normal uterine bleeding as bleeding occurring every 24–38 days (frequency), bleeding lasting no more than 8 days (duration), bleeding of a ‘normal’ amount as defined by the patient that does not interfere “with physical, social, emotional, and/or material quality of life” (volume), and bleeding within a menstrual cycle that only varies in length by plus or minus 4 days (regularity). FIGO further defines bleeding outside these normal parameters as abnormal uterine bleeding, which is divided into standard categories based on whether it is acute or chronic and the source or etiology of the abnormality according to the acronym PALM-COEIN (i.e., P olyp, A denomyosis, L eiomyoma, M alignancy and hyperplasia, C oagulopathy, O vulatory dysfunction, E ndometrial disorders, I atrogenic, and N ot otherwise classified). Other examples of efforts at standardization include menstrual hygiene indicators within the Water, Sanitation and Hygiene (WASH) field and defining how contraception can impact the menstrual cycle and analyzing these data in contraceptive studies [ 13 – 18 ].

Related to terminology, this review uses the phrase, “people who menstruate”, which we define as those who can menstruate, do menstruate, or have menstruated. Although people who menstruate may or may not identify as women or girls, and not all women and girls menstruate [ 19 ], we do use the terms ‘women’ and ‘girls’ in some instances, especially when citing primary literature and because menstrual health cannot “be adequately addressed without attention to the gender norms and dynamics experienced by individuals in the cultures and communities in which they live” [ 7 ]. As much as possible, however, we use gender inclusive terms and other people-first language.

Review scope

To aid in efforts for standardized measurement across the study of menstruation and the menstrual cycle, we systematically reviewed approaches to measure four aspects of changes to the menstrual cycle: bleeding, blood, pain, and perceptions of bleeding, blood, or pain. We use the term ‘ menstrual changes ’ to refer to these four aspects for the remainder of the paper. We sought to include all types of measures or methods for assessing menstrual changes (e.g., quantitative assays, biomarkers, data reported by clinicians, researchers, or directly by the person who menstruates). We use the term ‘ instruments ’ to refer to any of these measures or methods for the remainder of the paper. Our aim was to identify any instruments that have been developed and validated within any field of study to measure menstrual changes, examine how these instruments measured menstrual changes, and assess the measure quality of the identified instruments and their utility for the clinical trial context.

Related reviews have been conducted: (a) within fields such as menstrual hygiene or the study of heavy menstrual bleeding (HMB) [ 20 , 21 ]; (b) to measure single parameters like volume of menstrual blood loss [ 22 ]; and (c) for specific approaches like pictorial methods to diagnose HMB [ 23 ]. However, given the gaps and silos within menstrual health research, our aim was to conduct an expansive and transdisciplinary review to inform more standardized measurement across menstrual health research and clinical trials. For this reason, we sought to include menstrual changes caused by any etiology or source. There are many factors that can result in menstrual changes, including those endogenous and exogenous to the person who menstruates. Examples of these etiologies or sources include menstrual or gynecologic disorders (e.g., endometriosis, uterine fibroids, or adenomyosis), use of hormonal or intrauterine contraceptives, use of other drugs or devices to treat or prevent disease, environmental exposures, infectious disease, injury, coagulation disorders, and diet and exercise. We are not aware of any previous efforts to look at menstrual changes across disciplines in this way.

Clinical trial context

As mentioned, one area for which we intend our review to be quite relevant is for data collection in clinical trials, although our broad approach does not preclude the use of our results to inform the measurement of menstrual changes across other research contexts. The importance of data on menstrual changes in the clinical trial context was recently highlighted during the introduction of COVID-19 vaccinations. Because vaccine trials did not collect data on the impact to the menstrual cycle or menopausal uterine bleeding, there were concerns among vaccinated people who menstruate when they experienced these changes, which can erode trust in clinical research and public health interventions [ 24 – 28 ]. As authors working across various sexual and reproductive health spaces, our interest in conducting this review stemmed from a shared goal to improve and standardize the measurement of menstrual changes in contraceptive clinical trials; however, our broad methodological approach permits the utility of our findings across all clinical trials.

Clinical trials, and the preclinical research that precedes them, collect data on key organ functioning and vital signs as part of standard toxicology and pharmacodynamics. Given the importance of the menstrual cycle, it may seem surprising that data on how investigational drugs may impact the menstrual cycle are not already routinely collected in clinical trials; however, research typically reflect the people, priorities, and purposes of those within the clinical trial ecosystem—that is, the individuals and systems that fund clinical research, conduct clinical trials, and regulate the drugs tested in trials, as well as the individuals who participate in trials. Historically, there has been an underrepresentation of people who menstruate within the clinical trial ecosystem [ 29 ]. This exclusion is true for much of the preclinical research that informs clinical trials across many biomedical fields as well, and even cell lines used in in vitro studies are predominantly derived from male animals [ 30 , 31 ]. Although proof-of-concept studies for drugs intended for use in women that are known to impact the menstrual cycle, such as hormonal contraceptives, do typically use female animals when the model organism has an estrous or menstrual cycle, other preclinical research disproportionately relies on only male animals. Using both female and male animals in the research that informs clinical trials, however, could provide early indications of any impacts on cycles, as well as many other sex-specific effects or differences. Despite decades of concrete efforts, sex and gender disparities persist in the clinical trial ecosystem [ 32 – 34 ].

Within the current clinical trial context, another element relevant to our review is how trials typically incorporate outcomes, like menstrual changes, that are reported by trial participants. The United States Food and Drug Administration (FDA) and NIH refer to these data as patient-reported outcomes (PROs), which they define as “a measurement based on a report that comes directly from the patient (i.e., study subject) about the status of a patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else.” PROs can include “symptoms or other unobservable concepts known only to the patient (e.g., pain severity or nausea) [that] can only be measured by PRO measures,” as well as “the patient perspective on functioning or activities that may also be observable by others” [ 35 ]. Unless an assay or biomarker are used, all outcomes on menstrual changes are reported by the person who menstruates and, therefore, are PROs. The FDA has a series of methodological guidance documents on the development, validation, and use of PROs in clinical trials as part of patient-focused drug development efforts [ 36 – 39 ].

Review questions and objective

Given the aim of the review, our review questions were: (a) What instruments have been developed to assess menstrual changes caused by any etiology or source? and (b) What is the quality of these instruments and their utility for clinical trials? The objective of our systematic review was to compile a complete list of validated instruments used to measure menstrual changes with an assessment of their quality and clinical trial utility.

Materials and methods

We conducted our systematic review in alignment with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [ 40 – 42 ], including a protocol registered in PROSPERO (Protocol ID: CRD42023420358) [ 43 ]. A completed PRISMA checklist for this review is in S1 Table , and S1 Appendix includes additional details on the search strategy, inclusion/exclusion criteria, title/abstract screening, full text review, data extraction, and data analysis.

Search strategy

We searched for peer reviewed articles in the MEDLINE and Embase literature databases and for any relevant instruments measuring menstrual changes in four instrument databases: (a) the NIH Common Data Element (CDE) Repository [ 44 ], (b) the COSMIN database of systematic reviews of outcome measurement instruments [ 45 ], (c) the Core Outcome Measures in Effectiveness Trials (COMET) Database [ 46 ], and (c) ePROVIDE databases [ 47 ]. Table 1 shows the final search strategy for MEDLINE, and S1 Appendix includes search strategies for other databases. We uploaded articles from the literature database searches and articles for any relevant instruments identified via the instrument databases into Covidence [ 48 ]. Following screening and review of these articles in Covidence, we identified relevant review articles and extracted primary articles published since 1980 from those reviews. During data extraction, we identified any original articles for instruments developed before 2006. We uploaded these primary articles and original development articles into Covidence for screening.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0306491.t001

Inclusion/Exclusion criteria

We included all peer-reviewed articles—including those with prospective, retrospective, or cross-sectional study designs, and review papers—that met our inclusion and did not meet our exclusion criteria, listed in Table 2 .

https://doi.org/10.1371/journal.pone.0306491.t002

Title/Abstract screening, full text review, and data extraction

We held weekly author meetings to discuss progress, questions, and discordance, and to document decisions in a shared Word document. We began title/abstract screening with an ‘inter-reviewer reliability’ meeting where all authors completed title/abstract screening on the same 50 articles to establish and confirm group standards. Then, two authors independently screened each remaining title/abstract and two authors independently reviewed each relevant full text in Covidence. We resolved any discordance during weekly meetings via consensus conversations. We conducted data extraction in Excel using a template data extraction form that collected article information, study design, sample information, details on the instrument, measure quality attributes, and clinical trial utility attributes. For articles not in English, we used the text translation feature of Google Translate to review titles and abstracts, we used the document translation feature of Google Translate and/or consulted a fluent colleague to review full text articles, and we completed data extraction with a fluent colleague for included articles.

For assessing measure quality and clinical trial utility, we followed the recent Patient-Reported Outcomes Tools: Engaging Users and Stakeholders (PROTEUS) Consortium recommendations to use the International Society for Quality of Life Research (ISOQOL) standards for PRO measures [ 50 , 51 ]. We made two adjustments to the ISOQOL standards: (a) we added an attribute on sensitivity of questions given the topic of menstruation has a noted amount of stigma surrounding it [ 52 ]; and (b) we separated out participant burden from investigator burden given these two can differ greatly for instruments measuring menstrual changes. We categorized six attributes as related primarily to the quality of the instrument (i.e., measure quality : conceptual/measurement model, reliability, content validity, construct validity, responsiveness, and sensitive nature of questions) and four attributes as related primarily to the utility of the instrument in clinical trials (i.e., clinical trial utility : interpretability of results, the transferability of the instrument, participant burden, and investigator burden). We scored each attribute of measure quality and clinical trial utility on a scale from 0 to 3 (0 = no data, 1 = poor , 2 = fair , and 3 = good ) based on criteria in line with ISOQOL standards [ 51 ] that were reviewed by measurement and clinical experts at FHI 360 and within a related global task force. We show the measure quality and clinical trial utility attributes and scoring criteria in Table 3 , and S1 Appendix includes details on the other fields of the data extraction form.

https://doi.org/10.1371/journal.pone.0306491.t003

Data analysis

We conducted data analysis in Excel and included counts and frequencies, as well as specific analyses to assess instrument measure quality and clinical trial utility. For the measure quality score and clinical trial utility score of an instrument, we used an average of the highest score for each attribute of measure quality or clinical trial utility across all articles on an instrument. Because instruments could have more than one article providing data on measure quality and/or clinical trial utility and not every article evaluated all attributes of an instrument, we did not include scores of zero (i.e., no data reported) in these averages. To reflect these differences in the number of articles and attributes reported in the article(s), we also calculated a total evidence score , which was the total of all scores—including zeros—across all attributes of measure quality and clinical trial utility. The total evidence scores, therefore, ‘penalize’ instruments for a lower level of evidence due to fewer articles or less attribute data and vice versa.

These three scores—measure quality (ranging from 1–3), clinical trial utility (ranging from 1–3), and total evidence (ranging 0+)—reflect different dimensions of an instrument. For example, two instruments might both have a score of 2.5 for measure quality, but one instrument might have an evidence score of 10 and the other, 100, indicating the latter has considerably more evidence and likely more certainty in the measure quality score. Alternately, two instruments may have similar measure quality and evidence scores, but one may have a clinical trial utility score of 1 and the other a score of 3, indicating the latter is likely better suited for use in clinical trials despite the similar levels of measure quality and evidence.

Search results

Across databases, our searches yielded 8,490 articles. We removed 376 duplicates, excluded 7,704 articles during title and abstract screening, and excluded 236 articles during full text review. In total, we identified 174 relevant full text articles. We present additional details on our search results and screening in the PRISMA diagram in Fig 1 .

* Per Page et al ., 2021 [ 40 ].

https://doi.org/10.1371/journal.pone.0306491.g001

We found some similarities across papers that we excluded for not meeting our inclusion criteria. For example, we excluded conference presentations that never became full papers, studies that focused on validating instruments among only menopausal populations (e.g., [ 54 , 55 ]), and studies that only validated surgical or treatment outcomes (e.g., [ 56 , 57 ]). In addition, there were two recent papers on core outcome sets for HMB and endometriosis relevant to the wider topic of measuring changes to the menstrual cycle, but we were unable to include them because there were no instrument details to extract [ 58 , 59 ].

Included article characteristics

Over 85% of the 174 articles were from either Europe (43%), North America (32%) or Asia (13%), and there were less than 15 articles from South America (n = 13), from the Middle East (n = 11), from Oceania (n = 8) and from Africa (n = 5). Just under half of articles were from only the United States (28%) or the United Kingdom (16%), although we did identify articles from a total of 50 countries. Nearly all articles were in English—even those reporting on instruments in other languages—except for two in Portuguese [ 60 , 61 ]. The most common study designs were cross-sectional or prospective cohort. We present details of all 174 included articles in S2 Table .

Instrument characteristics

From the 174 included articles, we extracted 94 instruments. Almost three quarters (72%, n = 68) were full instruments, collecting data on one or more menstrual change. Nearly a quarter (n = 21) were broader instruments that included sub-scales (9%, n = 8) or a small number of items (14%, n = 13) on menstrual changes. Five percent (n = 5) were general instruments validated in menstruating populations on one or more menstrual change. The instruments with the most articles in our review were the Endometriosis Health Profile-30 (EHP-30; 20 articles), the Pictorial Blood Loss Assessment Charts & Menstrual Pictograms (PBAC; 11 articles), the Uterine Fibroid Symptom and Quality of Life questionnaire (UFS-QOL; 9 articles), the Polycystic Ovary Syndrome Quality of Life scale (PCOS-QOL; 8 articles), and the Endometriosis Health Profile-5 (EHP-5), Menstrual Attitudes Questionnaire (MAQ), and menstrual collection (5 articles each). About a third (38%, n = 26) of full instruments used electronic data collection, and almost all full instruments (97%, n = 66) were completed by only the patient/participant who menstruated (i.e., they were PROs per the FDA and NIH definition). We present the list of the 68 full instruments and instrument characteristics in Table 4 . The remainder of results reported below are for these full instruments, with details on the sub-scales, items, and general instruments in S3 Table .

https://doi.org/10.1371/journal.pone.0306491.t004

Language(s)

Of the 68 full instruments, two-thirds were in English (66%, n = 45), followed by Spanish (13%, n = 9), French (9%, n = 6), and Portuguese (9%, n = 6); however, we identified instruments in 28 languages. About forty percent of instruments (41%, n = 28) were only in English, although about a quarter of instruments (26%) were in more than one language, and six instruments were in at least 4 languages. These instruments included the EHP-30 (13 languages), UFS-QOL (5 languages), MAQ (5 languages), PBAC (4 languages), Endometriosis Daily Diary (EDD; 4 languages), and the Daily Diary (4 languages).

Specific populations

Nearly 60% (n = 40) of the 68 full instruments were developed and/or validated in populations with menstrual or gynecologic disorders or symptoms (i.e., 18 for endometriosis, 10 for HMB, 9 for dysmenorrhea, and 3 for uterine fibroids). Less than a quarter (24%, n = 16) of full instruments were developed for and validated with adolescents (mean ages less than 18, n = 10) or young people (mean ages early 20s, n = 6). Three full instruments were specifically developed for those in perimenopause. A few instruments were developed or validated in populations of athletes or people in the military. No instruments or articles indicated inclusion of trans and gender nonbinary people who menstruate.

Menstrual change(s) measured

Among the 68 full instruments, half (49%, n = 33) measured bleeding, nearly half (47%, n = 32) measured uterine cramping or pain, and almost three quarters (74%, n = 50) measured perceptions. Only eight (12%) measured blood characteristics. Three instruments assessed all four of the parameters of bleeding—duration, volume, frequency, and regularity/predictability (i.e., the Aberdeen Menorrhagia Severity Scale [AMSS], the New Zealand Survey of Adolescent Girls’ Menstruation, and the World Endometriosis Research Foundation Endometriosis Phenome and Biobanking Harmonisation Project Standard Questionnaire [WERF EPHect EPQ-S]). No instrument assessed all three parameters of blood—color, consistency, and smell.

Across the four aspects of menstrual changes (i.e., bleeding, blood, pain, and perceptions), no instrument measured all parameters for each aspect, and only seven instruments measured at least a single parameter of each aspect. These instruments were the AMSS; electronic Personal Assessment Questionnaire—Menstrual, Pain, and Hormonal (ePAQ-MPH); Endometriosis Self-Assessment Tool (ESAT); Fibroid Symptom Diary (FSD); Menstrual Bleeding Questionnaire (MBQ); Menstrual Insecurity Tool; and the New Zealand Survey of Adolescent Girls’ Menstruation.

How instruments measured menstrual changes

We present in Table 5 details on how the 68 full instruments measured bleeding (i.e., the four parameters of duration, volume, frequency, and regularity/predictability), blood (i.e., the three parameters of color, consistency, and smell), uterine cramping/pain, and perceptions.

https://doi.org/10.1371/journal.pone.0306491.t005

Measure quality of full instruments

When assessing measure quality, we found only five of the 68 full instruments (7%) had data on each of the six attributes of measure quality (i.e., conceptual or measurement model, reliability, content validity, construct validity, responsiveness, and sensitive nature of questions). These were the PBAC, EHP-30, Dysmenorrhea Daily Diary, MBQ, and a quantitative model for menstrual blood loss [ 85 ], each indicated by †† in Table 4 . All but three instruments (96%, n = 65) had evidence of a conceptual or measurement model and most also included evidence of content validity (81%, n = 55), construct validity (84%, n = 57) and reliability (66%, n = 45); however, less than a third of instruments had evidence on responsiveness (31%, n = 21), and less than a fifth (19%, n = 13) had evidence on question sensitivity ( Fig 2 ).

Of the 68 full instruments, 18% (n = 12) had an overall good measure quality score, about three quarters (74%, n = 50) had a fair measure quality score, and 9% (n = 6) had a poor measure quality score ( Table 4 and Fig 2). When we looked at individual attributes of measure quality, over half of instruments had a good score for content validity (56%, n = 38), 47% had a good score for reliability (n = 32), 44% had a good score for conceptual or measurement model (n = 30), and over a third of instruments (35%, n = 24) had a good score for construct validity; however, only a quarter had a good score for responsiveness (25%, n = 17), and only 4 instruments (6%) had a good score for question sensitivity.

https://doi.org/10.1371/journal.pone.0306491.g002

Utility for clinical trials of full instruments

When assessing clinical trial utility, we found 11 full instruments (16%) had data on each of the five attributes of utility (i.e., interpretability of results, transferability, participant burden, and investigator burden), each indicated by ‡ in Table 4 . All but three instruments (96%) had information on participant burden, 84% (n = 57) had evidence of the interpretability of the instrument results, and slightly less than two thirds (60%, n = 41) had documented investigator burden; however, only just over one third (37%, n = 25) had evidence of transferability ( Fig 3 ).

Of the 68 full instruments, 22% (n = 15) had an overall good clinical trial utility score, almost two thirds (62%, n = 42) had a fair score, and 13% (n = 9) had a poor score ( Table 4 and Fig 3). When we looked at individual attributes of clinical trial utility, almost half of instruments (49%, n = 33) had a good score for the interpretability of results, about 40% had good scores for participant burden (41%, n = 28) or investigator burden (40%, n = 27), but only 8 instruments (12%) had good scores for transferability.

https://doi.org/10.1371/journal.pone.0306491.g003

Overall full instrument evidence

Only the PBAC had evidence on all attributes of measure quality and all attributes of clinical trial utility, and only three instruments had both a good measure quality score and a good clinical trial utility score: EHP-5, the Spanish Society of Contraception Quality-of-Life (SEC-QOL), and the SAMANTA Questionnaire. Thirteen instruments had both measure quality scores and clinical trial utility scores greater than 2.5. Only one instrument, the Squeezing Pain Bulb, had both poor measure quality and poor clinical trial utility. Full instrument total evidence scores ranged from 4 for the World Health Organization Disability Assessment Schedule 2.0 to 332 for the EHP-30, with an overall median score across instruments of 16 and mean score of 27 ( Table 4 ). Overall, the following instruments had the five highest scores across measure quality, clinical trial utility, and total evidence: EHP-30, EHP-5, UFS-QOL, PBAC, and MBQ.

Our broad, transdisciplinary systematic review on the measurement of menstrual changes caused by any intrinsic or extrinsic factor, etiology, or source yielded 174 relevant articles and 94 instruments. Through our data extraction and analysis of these articles and instruments, we found several strengths and notable gaps in this literature around geographic and linguistic representation, how menstrual changes were measured, measure quality and clinical trial utility, and menstrual stigma, among others.

Geographic and linguistic representation

We identified articles from all geographic regions and 50 countries, and full instruments in 28 languages, including over a quarter in more than one language. Despite this evidence of the breadth of the literature, three quarters of articles were from North America or Europe and almost half were from just the United States and United Kingdom. In addition, over half of full instruments were only in English, Spanish, French, or Portuguese. These findings indicate the existing instrument landscape centers around the United States and Western Europe, as well as colonial languages.

How menstrual changes were measured

We again found promising strengths mixed with important gaps when examining the menstrual changes instruments measured and how they were measured. Although many full instruments measured perceptions and at least one parameter of bleeding or pain, only 8 full instruments measured blood. It is possible this lack of data collection on blood is due to the wide influence of menstrual stigma, especially the common perspective that menstrual blood is ‘dirty’ and requires ‘hygiene’ products to cleanse, absorb, and hide blood or odor [ 52 , 193 , 194 ]. No full instruments measured all parameters for each of the four aspects of menstrual changes we assessed, and only 7 instruments measured at least one parameter for all four aspects. In addition, across all aspects of menstrual changes, we did not find high levels of uniformity between instruments regarding how they measured each menstrual change, and many did not explain or define key terms (e.g., ‘heavy’, ‘regular’), leaving their interpretation up to each respondent. This lack of clarity and specificity raises concerns about measurement error for a topic like menstruation and the wider menstrual cycle, around which there is high stigma and low health literacy and therefore, reduced shared understanding and references. These findings indicate there is a lack of instruments that examine all parameters and aspects of changes to the menstrual cycle in a comprehensive and standardized way.

Nearly 60% of full instruments we identified were developed for those with menstrual or gynecologic disorders and symptoms. In fact, the 3 instruments that accounted for almost a quarter of all identified articles—the EHP-30, PBAC, and USF-QOL—were each developed for use in populations with endometriosis, HMB, and fibroids, respectively. Instruments for these populations are of crucial importance, and it is encouraging to see over 70% of identified articles published in the last 5 years examine menstrual or gynecologic disorders and symptoms. However, the measurement of menstrual changes resulting from these disorders, such as very heavy bleeding and high levels of pain, may not translate to the menstrual changes experienced by the wider menstruating population or to the range of menstrual changes likely to occur across clinical trials and related research. For example, the extension of an instrument developed for those with HMB to a clinical trial of a hormonal contraceptive—which generally decreases bleeding volume—is yet to be supported by evidence. This difference is important because we could hypothesize, for example, there would be a difference in recall from a bleeding episode that resulted in stained clothing (i.e., from HMB) compared to a bleeding episode that did not interfere with daily activities (i.e., from a hormonal contraceptive). Because of these findings, instruments likely need to be developed or modified to capture a wider array of changes in bleeding, blood, and pain, as well as changes that are of smaller—but still meaningful—magnitude.

Instrument quality and utility

From our assessments of measure quality and clinical trial utility for full instruments, we also found variability in our outcomes. Over 80% of instruments had either fair or good scores for measure quality or clinical trial utility, and only one had both poor measure quality and poor clinical trial utility. On the other hand, only three instruments had both good measure quality and good clinical trial utility.

We also note almost all instruments had evidence supporting some quality and utility attributes but not others. Sixty percent or more of instruments had evidence of a conceptual or measurement model, reliability, content validity, or construct validity for measure quality, or had evidence of interpretability of results, participant burden, or investigator burden for clinical trial utility; almost a quarter of instruments had evidence of each of these seven attributes. On the other hand, only one instrument—the PBAC—had evidence for all attributes of quality and utility, and over 60% of instruments did not have evidence of responsiveness, question sensitivity, or transferability, with nearly 40% not having evidence of any of the three. Each of these largely missing attributes are likely to be important for any instrument used broadly, especially in clinical trials. Such an instrument will need to: (a) capture changes during investigational drug use (responsiveness); (b) not be viewed as too intrusive or stigmatizing (question sensitivity); and (c) be used in multiple linguistic and sociocultural contexts (transferability).

Menstrual stigma and other notable gaps

Our findings on the limited measurement for blood and lack of evidence for question sensitivity highlight the importance of menstrual stigma. We often found a contradiction during the development and validation of instruments; although menstrual stigma was frequently acknowledged as part of the sociocultural milieu surrounding menstruation, instruments generally did not adequately address menstrual stigma or how stigma may relate to question sensitivity and the potential impact of this on data quality or measurement error.

Beyond the difficulty of measurement due to menstrual stigma, there is innate complexity in measuring changes to a biological process that, itself, consists of so many facets that change over time and vary between individuals [ 195 , 196 ]. For example, there are changes between days of a single menstrual cycle (e.g., different bleeding and/or pain experienced on different days of a cycle), differences among menstrual cycles during the same year, and shifts over the menstruating life course for one individual person who menstruates, as well as a multitude of differences between people [ 197 – 199 ]. These factors are important when we consider just under half of articles for the identified full instruments had cross-sectional study designs. In fact, this study design limitation could be the reason we found a lack of evidence on instrument responsiveness and measurement of more temporally-related parameters like bleeding frequency and regularity/predictability.

In addition to the gaps in the literature and instrument landscape already mentioned, three additional findings warrant attention. First, only just over a third of instruments used electronic data collection. Although this may be partly due to our review extending through 2006, new and refined instruments should strongly consider this approach given the data quality and monitoring benefits of electronic data collection and with the current proliferation of period tracking and other FemTech applications [ 200 , 201 ]. In addition, there is a need to establish the equivalence between existing paper instruments and any electronic versions developed, ideally in accordance with established approaches like the International Professional Society for Health Economics and Outcomes Research (ISPOR) good research practices on use of mixed mode PROs [ 202 ].

Second, there is a lack of attention paid to the two ends of the menstruating life course. There were only ten instruments specifically developed with data from adolescents and three instruments developed for those in perimenopause, both groups who can experience an increased amount of variability and change in their menstrual cycles as compared to the middle of the menstruating years [ 203 ]. In addition, data on older menstruators were often collapsed for people who were in perimenopause and menopause/post-menopause, or age was commonly used as a proxy for this process and transition. Although the age range for menopause is narrower than that of menarche, given the general lack of research around menopause and the preceding and succeeding years, it seems the opposite should be true (i.e., more data and larger sample sizes among people around the end of their menstruating years is warranted) [ 204 ].

Third, we found a lack of inclusion for trans and gender nonbinary populations in all articles for all instruments. As we note in the introduction of this paper, people who menstruate may or may not identify as women or girls, and not all women and girls menstruate. It is important to engage all populations who menstruate in the development of instruments to measure changes to the menstrual cycle. Inclusion of sexual and gender minority (SGM) individuals who menstruate in clinical trials is a noted priority among NIH and other funders and researchers. In addition to NIH establishing its SGM Research Office in 2015, clinical research is the first theme of the current Strategic Plan to Advance Research on the Health and Well-being of SGM populations [ 205 ].

Limitations of the review

Although we followed PRISMA guidelines and included ‘inter-reviewer reliability’ checks, weekly meetings, and multiple reviewers per article, there are a few limitations to note about our review process. The most important limitations are related to decisions made regarding the scope of the review to make it focused and feasible. First, we only included four aspects of changes to the menstrual cycle: changes in bleeding, blood, pain, and perceptions of bleeding, blood, or pain. Although these aspects are likely the most studied thus far, there are many other important changes to the menstrual cycle, including in hormone levels, the phases of the menstrual cycle, characteristics of those phases, and other symptoms besides pain. As the study of menstrual health grows, it will be important for future reviews to consider these areas of research. We also limited our scope to the menstrual cycle, excluding other types of uterine bleeding, such as bleeding during pregnancy, while breastfeeding, and after menopause. Future insights into how these types of bleeding relate to bleeding during the menstrual cycle will be important to the research and understanding of all uterine bleeding.

We also note a few limitations related to our review process. First, although all authors have training and experience across multiple disciplines, none are experts in all fields from which we drew our literature given our transdisciplinary approach. We aimed to address this limitation by consulting other experts internally at FHI 360 and members of a related global task force when we encountered a question or issue outside of our knowledgebase, but it is still possible we missed articles, data for extraction, or other elements due to this limitation. Second, the primary impetus for the review among the authors was to inform measurement of menstrual changes in the context of contraceptive clinical trials, so we cannot completely rule out the possibility this internal aim may have influenced our decisions about including or excluding articles. From the very beginning of the review, however, we sought for the review to be useful across contexts and disciplines, so our protocol and process were designed and implemented with that purpose in mind. Third, we may have missed articles by deciding to not include the CINAHL and PsycINFO databases in addition to those we did include (i.e., MEDLINE, Embase, and the instrument databases). Despite reviewing at least 50 articles most relevant to search strategies for CINAHL and PsycINFO and finding none aligned with our inclusion criteria, it is possible there were articles relevant to our review in the rest of the search results from these two databases. Fourth, because we did not want to exclude articles from any region or language but are not fluent in all languages, we used Google Translate for some screening and review. It is, therefore, possible this translation did not allow us to sufficiently evaluate articles per our inclusion and exclusion criteria. For the two relevant articles not written in English, we did complete data extraction with a fluent colleague. Overall, there may be additional limitations about which we are not aware that may have biased the results of our systematic review. Our hope is, however, we took steps to mitigate as many as possible by having our protocol and instrument evaluation criteria reviewed by other experts, following best practice guidelines, and taking steps to reduce individual variability and biases.

Despite the novel, broad, and transdisciplinary approach to our systematic review, the current instrument landscape, limitations in the literature, and gaps in evidence on measure quality and clinical trial utility indicate there is a need to examine changes to the menstrual cycle in a more complete, inclusive, and standardized way. Rigorous formative research—across sociocultural contexts—that is focused on how all people who menstruate experience and understand their menstrual cycles and more fully addresses menstrual stigma can inform the development of new or modified instruments to meet this need. We also identified a need for greater evidence of the validity for existing and new instruments. For the clinical trial context, FDA guidance on selecting, developing, or modifying patient-reported outcomes like menstrual changes indicate there must be evidence to support the use of an instrument for the specific concepts of interest and context of use [ 37 ]. At a minimum, per this guidance, evidence would be needed to support the use of the instruments identified and assessed in this review in the clinical trial context with a broader patient population (i.e., context of use) and to measure the full scope of menstrual changes that people experience (i.e., concept of interest). In addition, the recent emergence of core outcome sets within areas like HMB and endometriosis will be useful to promote standardization of validated instruments, especially if these efforts are interdisciplinary and coordinated across research areas.

The findings of our review will be helpful in developing new or modified instruments that assess menstrual changes in a validated, comprehensive way. If used across the many fields that study menstrual health, data from these standardized instruments can contribute to an interdisciplinary, systemic, and holistic understanding of menstruation and the menstrual cycle. In turn, this improved understanding can be translated into ways to enhance the health and wellbeing of people who menstruate.

Supporting information

S1 table. preferred reporting items for systematic reviews and meta-analysis (prisma) checklist..

https://doi.org/10.1371/journal.pone.0306491.s001

S2 Table. All articles included after title/abstract screening and full text review.

https://doi.org/10.1371/journal.pone.0306491.s002

S3 Table. Characteristics of sub-scales, items, and general instruments.

https://doi.org/10.1371/journal.pone.0306491.s003

S1 Appendix. Details on review methods.

https://doi.org/10.1371/journal.pone.0306491.s004

Acknowledgments

The authors would like to thank Laneta Dorflinger, Rebecca Callahan, and members of the Global Contraceptive Induced Menstrual Changes (CIMC) Task Force for their feedback on the draft review protocol. We are very grateful for the guidance and assistance provided by the FHI 360 health sciences library throughout the development and implementation of our literature search strategy (Allison Burns and Carol Manion) and during retrieval of full text articles (Tamara Fasnacht). We would also like to acknowledge Betsy Costenbader, Kavita Nanda, and Global CIMC Task Force members for their feedback on the draft data extraction forms, and Kavita Nanda for clinical advice. We would also like to thank Valeria Bahamondes for her assistance in the full text review and data extraction of the two Portuguese papers that were included. The authors also appreciate Laneta Dorflinger and Kate McQueen for their review of and feedback on drafts of this manuscript.

• View Article
• Google Scholar
• PubMed/NCBI
• 6. Bobel C, Winkler IT, Fahs B, Hasson KA, Kissling EA, Roberts T-A, editors. The Palgrave Handbook of Critical Menstruation Studies. Singapore: Springer Singapore; 2020. https://doi.org/10.1007/978-981-15-0614-7
• 8. Guilló-Arakistain M. Challenging Menstrual Normativity: Nonessentialist Body Politics and Feminist Epistemologies of Health. In: Bobel C, Winkler I, Fahs B, Hasson K, Kissling EarT, editors. The Palgrave Handbook of Critical Menstruation Studies. Singapore: Springer Singapore; 2020. pp. 869–883. https://doi.org/10.1007/978-981-15-0614-7_63
• 13. WHO/UNICEF. Consultation on draft long list of goal, target and indicator options for future global monitoring of water, sanitation and hygiene. [cited 24 Jul 2023]. Available: https://washdata.org/sites/default/files/documents/reports/2017-06/JMP-2012-post2015-consultation.pdf .
• 19. Bobel C. The Managed Body: Developing Girls and Menstrual Health in the Global South. Cham, Switzerland: Palgrave Macmillan, Springer Nature; 2019. https://doi.org/10.1007/978-3-319-89414-0
• 44. US National Library of Medicine, US National Institutes of Health. NIH Common Data Elements Repository. [cited 11 Oct 2023]. Available: https://cde.nlm.nih.gov .
• 45. University Library Vrije Universiteit Amsterdam. COSMIN database of systematic reviews of outcome measurement instruments. [cited 11 Oct 2023]. Available: https://database.cosmin.nl .
• 46. Core Outcome Measures in Effectiveness Trials (COMET) Initiative. COMET database. [cited 11 Oct 2023]. Available: https://www.comet-initiative.org/Studies .
• 47. Mapi Research Trust. ePROVIDE. [cited 11 Oct 2023]. Available: https://eprovide.mapi-trust.org .
• 48. Veritas Health Innovation. Covidence Systematic Review Software. [cited 24 Jul 2023]. Available: https://www.covidence.org .
• 49. de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in Medicine. Cambridge University Press; 2011. https://doi.org/10.1017/CBO9780511996214
• 52. Johnston-Robledo I, Chrisler JC. The Menstrual Mark: Menstruation as Social Stigma. In: Bobel C, Winkler I, Fahs B, Hasson K, Kissling E, Roberts T, editors. The Palgrave Handbook of Critical Menstruation Studies. Singapore: Springer Singapore; 2020. pp. 181–199. https://doi.org/10.1007/978-981-15-0614-7_17
• 195. Clancy K. Period: The Real Story of Menstruation. Princeton, New Jersey, US: Princeton University Press; 2023.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.25(3); 2014 Oct

Peer Review in Scientific Publications: Benefits, Critiques, & A Survival Guide

Jacalyn kelly.

1 Clinical Biochemistry, Department of Pediatric Laboratory Medicine, The Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada

Tara Sadeghieh

Khosrow adeli.

2 Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada

3 Chair, Communications and Publications Division (CPD), International Federation for Sick Clinical Chemistry (IFCC), Milan, Italy

The authors declare no conflicts of interest regarding publication of this article.

Peer review has been defined as a process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field. It functions to encourage authors to meet the accepted high standards of their discipline and to control the dissemination of research data to ensure that unwarranted claims, unacceptable interpretations or personal views are not published without prior expert review. Despite its wide-spread use by most journals, the peer review process has also been widely criticised due to the slowness of the process to publish new findings and due to perceived bias by the editors and/or reviewers. Within the scientific community, peer review has become an essential component of the academic writing process. It helps ensure that papers published in scientific journals answer meaningful research questions and draw accurate conclusions based on professionally executed experimentation. Submission of low quality manuscripts has become increasingly prevalent, and peer review acts as a filter to prevent this work from reaching the scientific community. The major advantage of a peer review process is that peer-reviewed articles provide a trusted form of scientific communication. Since scientific knowledge is cumulative and builds on itself, this trust is particularly important. Despite the positive impacts of peer review, critics argue that the peer review process stifles innovation in experimentation, and acts as a poor screen against plagiarism. Despite its downfalls, there has not yet been a foolproof system developed to take the place of peer review, however, researchers have been looking into electronic means of improving the peer review process. Unfortunately, the recent explosion in online only/electronic journals has led to mass publication of a large number of scientific articles with little or no peer review. This poses significant risk to advances in scientific knowledge and its future potential. The current article summarizes the peer review process, highlights the pros and cons associated with different types of peer review, and describes new methods for improving peer review.

WHAT IS PEER REVIEW AND WHAT IS ITS PURPOSE?

Peer Review is defined as “a process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field” ( 1 ). Peer review is intended to serve two primary purposes. Firstly, it acts as a filter to ensure that only high quality research is published, especially in reputable journals, by determining the validity, significance and originality of the study. Secondly, peer review is intended to improve the quality of manuscripts that are deemed suitable for publication. Peer reviewers provide suggestions to authors on how to improve the quality of their manuscripts, and also identify any errors that need correcting before publication.

HISTORY OF PEER REVIEW

The concept of peer review was developed long before the scholarly journal. In fact, the peer review process is thought to have been used as a method of evaluating written work since ancient Greece ( 2 ). The peer review process was first described by a physician named Ishaq bin Ali al-Rahwi of Syria, who lived from 854-931 CE, in his book Ethics of the Physician ( 2 ). There, he stated that physicians must take notes describing the state of their patients’ medical conditions upon each visit. Following treatment, the notes were scrutinized by a local medical council to determine whether the physician had met the required standards of medical care. If the medical council deemed that the appropriate standards were not met, the physician in question could receive a lawsuit from the maltreated patient ( 2 ).

The invention of the printing press in 1453 allowed written documents to be distributed to the general public ( 3 ). At this time, it became more important to regulate the quality of the written material that became publicly available, and editing by peers increased in prevalence. In 1620, Francis Bacon wrote the work Novum Organum, where he described what eventually became known as the first universal method for generating and assessing new science ( 3 ). His work was instrumental in shaping the Scientific Method ( 3 ). In 1665, the French Journal des sçavans and the English Philosophical Transactions of the Royal Society were the first scientific journals to systematically publish research results ( 4 ). Philosophical Transactions of the Royal Society is thought to be the first journal to formalize the peer review process in 1665 ( 5 ), however, it is important to note that peer review was initially introduced to help editors decide which manuscripts to publish in their journals, and at that time it did not serve to ensure the validity of the research ( 6 ). It did not take long for the peer review process to evolve, and shortly thereafter papers were distributed to reviewers with the intent of authenticating the integrity of the research study before publication. The Royal Society of Edinburgh adhered to the following peer review process, published in their Medical Essays and Observations in 1731: “Memoirs sent by correspondence are distributed according to the subject matter to those members who are most versed in these matters. The report of their identity is not known to the author.” ( 7 ). The Royal Society of London adopted this review procedure in 1752 and developed the “Committee on Papers” to review manuscripts before they were published in Philosophical Transactions ( 6 ).

Peer review in the systematized and institutionalized form has developed immensely since the Second World War, at least partly due to the large increase in scientific research during this period ( 7 ). It is now used not only to ensure that a scientific manuscript is experimentally and ethically sound, but also to determine which papers sufficiently meet the journal’s standards of quality and originality before publication. Peer review is now standard practice by most credible scientific journals, and is an essential part of determining the credibility and quality of work submitted.

IMPACT OF THE PEER REVIEW PROCESS

Peer review has become the foundation of the scholarly publication system because it effectively subjects an author’s work to the scrutiny of other experts in the field. Thus, it encourages authors to strive to produce high quality research that will advance the field. Peer review also supports and maintains integrity and authenticity in the advancement of science. A scientific hypothesis or statement is generally not accepted by the academic community unless it has been published in a peer-reviewed journal ( 8 ). The Institute for Scientific Information ( ISI ) only considers journals that are peer-reviewed as candidates to receive Impact Factors. Peer review is a well-established process which has been a formal part of scientific communication for over 300 years.

OVERVIEW OF THE PEER REVIEW PROCESS

The peer review process begins when a scientist completes a research study and writes a manuscript that describes the purpose, experimental design, results, and conclusions of the study. The scientist then submits this paper to a suitable journal that specializes in a relevant research field, a step referred to as pre-submission. The editors of the journal will review the paper to ensure that the subject matter is in line with that of the journal, and that it fits with the editorial platform. Very few papers pass this initial evaluation. If the journal editors feel the paper sufficiently meets these requirements and is written by a credible source, they will send the paper to accomplished researchers in the field for a formal peer review. Peer reviewers are also known as referees (this process is summarized in Figure 1 ). The role of the editor is to select the most appropriate manuscripts for the journal, and to implement and monitor the peer review process. Editors must ensure that peer reviews are conducted fairly, and in an effective and timely manner. They must also ensure that there are no conflicts of interest involved in the peer review process.

Overview of the review process

When a reviewer is provided with a paper, he or she reads it carefully and scrutinizes it to evaluate the validity of the science, the quality of the experimental design, and the appropriateness of the methods used. The reviewer also assesses the significance of the research, and judges whether the work will contribute to advancement in the field by evaluating the importance of the findings, and determining the originality of the research. Additionally, reviewers identify any scientific errors and references that are missing or incorrect. Peer reviewers give recommendations to the editor regarding whether the paper should be accepted, rejected, or improved before publication in the journal. The editor will mediate author-referee discussion in order to clarify the priority of certain referee requests, suggest areas that can be strengthened, and overrule reviewer recommendations that are beyond the study’s scope ( 9 ). If the paper is accepted, as per suggestion by the peer reviewer, the paper goes into the production stage, where it is tweaked and formatted by the editors, and finally published in the scientific journal. An overview of the review process is presented in Figure 1 .

WHO CONDUCTS REVIEWS?

Peer reviews are conducted by scientific experts with specialized knowledge on the content of the manuscript, as well as by scientists with a more general knowledge base. Peer reviewers can be anyone who has competence and expertise in the subject areas that the journal covers. Reviewers can range from young and up-and-coming researchers to old masters in the field. Often, the young reviewers are the most responsive and deliver the best quality reviews, though this is not always the case. On average, a reviewer will conduct approximately eight reviews per year, according to a study on peer review by the Publishing Research Consortium (PRC) ( 7 ). Journals will often have a pool of reviewers with diverse backgrounds to allow for many different perspectives. They will also keep a rather large reviewer bank, so that reviewers do not get burnt out, overwhelmed or time constrained from reviewing multiple articles simultaneously.

WHY DO REVIEWERS REVIEW?

Referees are typically not paid to conduct peer reviews and the process takes considerable effort, so the question is raised as to what incentive referees have to review at all. Some feel an academic duty to perform reviews, and are of the mentality that if their peers are expected to review their papers, then they should review the work of their peers as well. Reviewers may also have personal contacts with editors, and may want to assist as much as possible. Others review to keep up-to-date with the latest developments in their field, and reading new scientific papers is an effective way to do so. Some scientists use peer review as an opportunity to advance their own research as it stimulates new ideas and allows them to read about new experimental techniques. Other reviewers are keen on building associations with prestigious journals and editors and becoming part of their community, as sometimes reviewers who show dedication to the journal are later hired as editors. Some scientists see peer review as a chance to become aware of the latest research before their peers, and thus be first to develop new insights from the material. Finally, in terms of career development, peer reviewing can be desirable as it is often noted on one’s resume or CV. Many institutions consider a researcher’s involvement in peer review when assessing their performance for promotions ( 11 ). Peer reviewing can also be an effective way for a scientist to show their superiors that they are committed to their scientific field ( 5 ).

ARE REVIEWERS KEEN TO REVIEW?

A 2009 international survey of 4000 peer reviewers conducted by the charity Sense About Science at the British Science Festival at the University of Surrey, found that 90% of reviewers were keen to peer review ( 12 ). One third of respondents to the survey said they were happy to review up to five papers per year, and an additional one third of respondents were happy to review up to ten.

HOW LONG DOES IT TAKE TO REVIEW ONE PAPER?

On average, it takes approximately six hours to review one paper ( 12 ), however, this number may vary greatly depending on the content of the paper and the nature of the peer reviewer. One in every 100 participants in the “Sense About Science” survey claims to have taken more than 100 hours to review their last paper ( 12 ).

HOW TO DETERMINE IF A JOURNAL IS PEER REVIEWED

Ulrichsweb is a directory that provides information on over 300,000 periodicals, including information regarding which journals are peer reviewed ( 13 ). After logging into the system using an institutional login (eg. from the University of Toronto), search terms, journal titles or ISSN numbers can be entered into the search bar. The database provides the title, publisher, and country of origin of the journal, and indicates whether the journal is still actively publishing. The black book symbol (labelled ‘refereed’) reveals that the journal is peer reviewed.

THE EVALUATION CRITERIA FOR PEER REVIEW OF SCIENTIFIC PAPERS

As previously mentioned, when a reviewer receives a scientific manuscript, he/she will first determine if the subject matter is well suited for the content of the journal. The reviewer will then consider whether the research question is important and original, a process which may be aided by a literature scan of review articles.

Scientific papers submitted for peer review usually follow a specific structure that begins with the title, followed by the abstract, introduction, methodology, results, discussion, conclusions, and references. The title must be descriptive and include the concept and organism investigated, and potentially the variable manipulated and the systems used in the study. The peer reviewer evaluates if the title is descriptive enough, and ensures that it is clear and concise. A study by the National Association of Realtors (NAR) published by the Oxford University Press in 2006 indicated that the title of a manuscript plays a significant role in determining reader interest, as 72% of respondents said they could usually judge whether an article will be of interest to them based on the title and the author, while 13% of respondents claimed to always be able to do so ( 14 ).

The abstract is a summary of the paper, which briefly mentions the background or purpose, methods, key results, and major conclusions of the study. The peer reviewer assesses whether the abstract is sufficiently informative and if the content of the abstract is consistent with the rest of the paper. The NAR study indicated that 40% of respondents could determine whether an article would be of interest to them based on the abstract alone 60-80% of the time, while 32% could judge an article based on the abstract 80-100% of the time ( 14 ). This demonstrates that the abstract alone is often used to assess the value of an article.

The introduction of a scientific paper presents the research question in the context of what is already known about the topic, in order to identify why the question being studied is of interest to the scientific community, and what gap in knowledge the study aims to fill ( 15 ). The introduction identifies the study’s purpose and scope, briefly describes the general methods of investigation, and outlines the hypothesis and predictions ( 15 ). The peer reviewer determines whether the introduction provides sufficient background information on the research topic, and ensures that the research question and hypothesis are clearly identifiable.

The methods section describes the experimental procedures, and explains why each experiment was conducted. The methods section also includes the equipment and reagents used in the investigation. The methods section should be detailed enough that it can be used it to repeat the experiment ( 15 ). Methods are written in the past tense and in the active voice. The peer reviewer assesses whether the appropriate methods were used to answer the research question, and if they were written with sufficient detail. If information is missing from the methods section, it is the peer reviewer’s job to identify what details need to be added.

The results section is where the outcomes of the experiment and trends in the data are explained without judgement, bias or interpretation ( 15 ). This section can include statistical tests performed on the data, as well as figures and tables in addition to the text. The peer reviewer ensures that the results are described with sufficient detail, and determines their credibility. Reviewers also confirm that the text is consistent with the information presented in tables and figures, and that all figures and tables included are important and relevant ( 15 ). The peer reviewer will also make sure that table and figure captions are appropriate both contextually and in length, and that tables and figures present the data accurately.

The discussion section is where the data is analyzed. Here, the results are interpreted and related to past studies ( 15 ). The discussion describes the meaning and significance of the results in terms of the research question and hypothesis, and states whether the hypothesis was supported or rejected. This section may also provide possible explanations for unusual results and suggestions for future research ( 15 ). The discussion should end with a conclusions section that summarizes the major findings of the investigation. The peer reviewer determines whether the discussion is clear and focused, and whether the conclusions are an appropriate interpretation of the results. Reviewers also ensure that the discussion addresses the limitations of the study, any anomalies in the results, the relationship of the study to previous research, and the theoretical implications and practical applications of the study.

The references are found at the end of the paper, and list all of the information sources cited in the text to describe the background, methods, and/or interpret results. Depending on the citation method used, the references are listed in alphabetical order according to author last name, or numbered according to the order in which they appear in the paper. The peer reviewer ensures that references are used appropriately, cited accurately, formatted correctly, and that none are missing.

Finally, the peer reviewer determines whether the paper is clearly written and if the content seems logical. After thoroughly reading through the entire manuscript, they determine whether it meets the journal’s standards for publication,

and whether it falls within the top 25% of papers in its field ( 16 ) to determine priority for publication. An overview of what a peer reviewer looks for when evaluating a manuscript, in order of importance, is presented in Figure 2 .

How a peer review evaluates a manuscript

To increase the chance of success in the peer review process, the author must ensure that the paper fully complies with the journal guidelines before submission. The author must also be open to criticism and suggested revisions, and learn from mistakes made in previous submissions.

ADVANTAGES AND DISADVANTAGES OF THE DIFFERENT TYPES OF PEER REVIEW

The peer review process is generally conducted in one of three ways: open review, single-blind review, or double-blind review. In an open review, both the author of the paper and the peer reviewer know one another’s identity. Alternatively, in single-blind review, the reviewer’s identity is kept private, but the author’s identity is revealed to the reviewer. In double-blind review, the identities of both the reviewer and author are kept anonymous. Open peer review is advantageous in that it prevents the reviewer from leaving malicious comments, being careless, or procrastinating completion of the review ( 2 ). It encourages reviewers to be open and honest without being disrespectful. Open reviewing also discourages plagiarism amongst authors ( 2 ). On the other hand, open peer review can also prevent reviewers from being honest for fear of developing bad rapport with the author. The reviewer may withhold or tone down their criticisms in order to be polite ( 2 ). This is especially true when younger reviewers are given a more esteemed author’s work, in which case the reviewer may be hesitant to provide criticism for fear that it will damper their relationship with a superior ( 2 ). According to the Sense About Science survey, editors find that completely open reviewing decreases the number of people willing to participate, and leads to reviews of little value ( 12 ). In the aforementioned study by the PRC, only 23% of authors surveyed had experience with open peer review ( 7 ).

Single-blind peer review is by far the most common. In the PRC study, 85% of authors surveyed had experience with single-blind peer review ( 7 ). This method is advantageous as the reviewer is more likely to provide honest feedback when their identity is concealed ( 2 ). This allows the reviewer to make independent decisions without the influence of the author ( 2 ). The main disadvantage of reviewer anonymity, however, is that reviewers who receive manuscripts on subjects similar to their own research may be tempted to delay completing the review in order to publish their own data first ( 2 ).

Double-blind peer review is advantageous as it prevents the reviewer from being biased against the author based on their country of origin or previous work ( 2 ). This allows the paper to be judged based on the quality of the content, rather than the reputation of the author. The Sense About Science survey indicates that 76% of researchers think double-blind peer review is a good idea ( 12 ), and the PRC survey indicates that 45% of authors have had experience with double-blind peer review ( 7 ). The disadvantage of double-blind peer review is that, especially in niche areas of research, it can sometimes be easy for the reviewer to determine the identity of the author based on writing style, subject matter or self-citation, and thus, impart bias ( 2 ).

Masking the author’s identity from peer reviewers, as is the case in double-blind review, is generally thought to minimize bias and maintain review quality. A study by Justice et al. in 1998 investigated whether masking author identity affected the quality of the review ( 17 ). One hundred and eighteen manuscripts were randomized; 26 were peer reviewed as normal, and 92 were moved into the ‘intervention’ arm, where editor quality assessments were completed for 77 manuscripts and author quality assessments were completed for 40 manuscripts ( 17 ). There was no perceived difference in quality between the masked and unmasked reviews. Additionally, the masking itself was often unsuccessful, especially with well-known authors ( 17 ). However, a previous study conducted by McNutt et al. had different results ( 18 ). In this case, blinding was successful 73% of the time, and they found that when author identity was masked, the quality of review was slightly higher ( 18 ). Although Justice et al. argued that this difference was too small to be consequential, their study targeted only biomedical journals, and the results cannot be generalized to journals of a different subject matter ( 17 ). Additionally, there were problems masking the identities of well-known authors, introducing a flaw in the methods. Regardless, Justice et al. concluded that masking author identity from reviewers may not improve review quality ( 17 ).

In addition to open, single-blind and double-blind peer review, there are two experimental forms of peer review. In some cases, following publication, papers may be subjected to post-publication peer review. As many papers are now published online, the scientific community has the opportunity to comment on these papers, engage in online discussions and post a formal review. For example, online publishers PLOS and BioMed Central have enabled scientists to post comments on published papers if they are registered users of the site ( 10 ). Philica is another journal launched with this experimental form of peer review. Only 8% of authors surveyed in the PRC study had experience with post-publication review ( 7 ). Another experimental form of peer review called Dynamic Peer Review has also emerged. Dynamic peer review is conducted on websites such as Naboj, which allow scientists to conduct peer reviews on articles in the preprint media ( 19 ). The peer review is conducted on repositories and is a continuous process, which allows the public to see both the article and the reviews as the article is being developed ( 19 ). Dynamic peer review helps prevent plagiarism as the scientific community will already be familiar with the work before the peer reviewed version appears in print ( 19 ). Dynamic review also reduces the time lag between manuscript submission and publishing. An example of a preprint server is the ‘arXiv’ developed by Paul Ginsparg in 1991, which is used primarily by physicists ( 19 ). These alternative forms of peer review are still un-established and experimental. Traditional peer review is time-tested and still highly utilized. All methods of peer review have their advantages and deficiencies, and all are prone to error.

PEER REVIEW OF OPEN ACCESS JOURNALS

Open access (OA) journals are becoming increasingly popular as they allow the potential for widespread distribution of publications in a timely manner ( 20 ). Nevertheless, there can be issues regarding the peer review process of open access journals. In a study published in Science in 2013, John Bohannon submitted 304 slightly different versions of a fictional scientific paper (written by a fake author, working out of a non-existent institution) to a selected group of OA journals. This study was performed in order to determine whether papers submitted to OA journals are properly reviewed before publication in comparison to subscription-based journals. The journals in this study were selected from the Directory of Open Access Journals (DOAJ) and Biall’s List, a list of journals which are potentially predatory, and all required a fee for publishing ( 21 ). Of the 304 journals, 157 accepted a fake paper, suggesting that acceptance was based on financial interest rather than the quality of article itself, while 98 journals promptly rejected the fakes ( 21 ). Although this study highlights useful information on the problems associated with lower quality publishers that do not have an effective peer review system in place, the article also generalizes the study results to all OA journals, which can be detrimental to the general perception of OA journals. There were two limitations of the study that made it impossible to accurately determine the relationship between peer review and OA journals: 1) there was no control group (subscription-based journals), and 2) the fake papers were sent to a non-randomized selection of journals, resulting in bias.

JOURNAL ACCEPTANCE RATES

Based on a recent survey, the average acceptance rate for papers submitted to scientific journals is about 50% ( 7 ). Twenty percent of the submitted manuscripts that are not accepted are rejected prior to review, and 30% are rejected following review ( 7 ). Of the 50% accepted, 41% are accepted with the condition of revision, while only 9% are accepted without the request for revision ( 7 ).

SATISFACTION WITH THE PEER REVIEW SYSTEM

Based on a recent survey by the PRC, 64% of academics are satisfied with the current system of peer review, and only 12% claimed to be ‘dissatisfied’ ( 7 ). The large majority, 85%, agreed with the statement that ‘scientific communication is greatly helped by peer review’ ( 7 ). There was a similarly high level of support (83%) for the idea that peer review ‘provides control in scientific communication’ ( 7 ).

HOW TO PEER REVIEW EFFECTIVELY

The following are ten tips on how to be an effective peer reviewer as indicated by Brian Lucey, an expert on the subject ( 22 ):

1) Be professional

Peer review is a mutual responsibility among fellow scientists, and scientists are expected, as part of the academic community, to take part in peer review. If one is to expect others to review their work, they should commit to reviewing the work of others as well, and put effort into it.

2) Be pleasant

If the paper is of low quality, suggest that it be rejected, but do not leave ad hominem comments. There is no benefit to being ruthless.

3) Read the invite

When emailing a scientist to ask them to conduct a peer review, the majority of journals will provide a link to either accept or reject. Do not respond to the email, respond to the link.

4) Be helpful

Suggest how the authors can overcome the shortcomings in their paper. A review should guide the author on what is good and what needs work from the reviewer’s perspective.

5) Be scientific

The peer reviewer plays the role of a scientific peer, not an editor for proofreading or decision-making. Don’t fill a review with comments on editorial and typographic issues. Instead, focus on adding value with scientific knowledge and commenting on the credibility of the research conducted and conclusions drawn. If the paper has a lot of typographical errors, suggest that it be professionally proof edited as part of the review.

6) Be timely

Stick to the timeline given when conducting a peer review. Editors track who is reviewing what and when and will know if someone is late on completing a review. It is important to be timely both out of respect for the journal and the author, as well as to not develop a reputation of being late for review deadlines.

7) Be realistic

The peer reviewer must be realistic about the work presented, the changes they suggest and their role. Peer reviewers may set the bar too high for the paper they are editing by proposing changes that are too ambitious and editors must override them.

8) Be empathetic

Ensure that the review is scientific, helpful and courteous. Be sensitive and respectful with word choice and tone in a review.

Remember that both specialists and generalists can provide valuable insight when peer reviewing. Editors will try to get both specialised and general reviewers for any particular paper to allow for different perspectives. If someone is asked to review, the editor has determined they have a valid and useful role to play, even if the paper is not in their area of expertise.

10) Be organised

A review requires structure and logical flow. A reviewer should proofread their review before submitting it for structural, grammatical and spelling errors as well as for clarity. Most publishers provide short guides on structuring a peer review on their website. Begin with an overview of the proposed improvements; then provide feedback on the paper structure, the quality of data sources and methods of investigation used, the logical flow of argument, and the validity of conclusions drawn. Then provide feedback on style, voice and lexical concerns, with suggestions on how to improve.

In addition, the American Physiology Society (APS) recommends in its Peer Review 101 Handout that peer reviewers should put themselves in both the editor’s and author’s shoes to ensure that they provide what both the editor and the author need and expect ( 11 ). To please the editor, the reviewer should ensure that the peer review is completed on time, and that it provides clear explanations to back up recommendations. To be helpful to the author, the reviewer must ensure that their feedback is constructive. It is suggested that the reviewer take time to think about the paper; they should read it once, wait at least a day, and then re-read it before writing the review ( 11 ). The APS also suggests that Graduate students and researchers pay attention to how peer reviewers edit their work, as well as to what edits they find helpful, in order to learn how to peer review effectively ( 11 ). Additionally, it is suggested that Graduate students practice reviewing by editing their peers’ papers and asking a faculty member for feedback on their efforts. It is recommended that young scientists offer to peer review as often as possible in order to become skilled at the process ( 11 ). The majority of students, fellows and trainees do not get formal training in peer review, but rather learn by observing their mentors. According to the APS, one acquires experience through networking and referrals, and should therefore try to strengthen relationships with journal editors by offering to review manuscripts ( 11 ). The APS also suggests that experienced reviewers provide constructive feedback to students and junior colleagues on their peer review efforts, and encourages them to peer review to demonstrate the importance of this process in improving science ( 11 ).

The peer reviewer should only comment on areas of the manuscript that they are knowledgeable about ( 23 ). If there is any section of the manuscript they feel they are not qualified to review, they should mention this in their comments and not provide further feedback on that section. The peer reviewer is not permitted to share any part of the manuscript with a colleague (even if they may be more knowledgeable in the subject matter) without first obtaining permission from the editor ( 23 ). If a peer reviewer comes across something they are unsure of in the paper, they can consult the literature to try and gain insight. It is important for scientists to remember that if a paper can be improved by the expertise of one of their colleagues, the journal must be informed of the colleague’s help, and approval must be obtained for their colleague to read the protected document. Additionally, the colleague must be identified in the confidential comments to the editor, in order to ensure that he/she is appropriately credited for any contributions ( 23 ). It is the job of the reviewer to make sure that the colleague assisting is aware of the confidentiality of the peer review process ( 23 ). Once the review is complete, the manuscript must be destroyed and cannot be saved electronically by the reviewers ( 23 ).

COMMON ERRORS IN SCIENTIFIC PAPERS

When performing a peer review, there are some common scientific errors to look out for. Most of these errors are violations of logic and common sense: these may include contradicting statements, unwarranted conclusions, suggestion of causation when there is only support for correlation, inappropriate extrapolation, circular reasoning, or pursuit of a trivial question ( 24 ). It is also common for authors to suggest that two variables are different because the effects of one variable are statistically significant while the effects of the other variable are not, rather than directly comparing the two variables ( 24 ). Authors sometimes oversee a confounding variable and do not control for it, or forget to include important details on how their experiments were controlled or the physical state of the organisms studied ( 24 ). Another common fault is the author’s failure to define terms or use words with precision, as these practices can mislead readers ( 24 ). Jargon and/or misused terms can be a serious problem in papers. Inaccurate statements about specific citations are also a common occurrence ( 24 ). Additionally, many studies produce knowledge that can be applied to areas of science outside the scope of the original study, therefore it is better for reviewers to look at the novelty of the idea, conclusions, data, and methodology, rather than scrutinize whether or not the paper answered the specific question at hand ( 24 ). Although it is important to recognize these points, when performing a review it is generally better practice for the peer reviewer to not focus on a checklist of things that could be wrong, but rather carefully identify the problems specific to each paper and continuously ask themselves if anything is missing ( 24 ). An extremely detailed description of how to conduct peer review effectively is presented in the paper How I Review an Original Scientific Article written by Frederic G. Hoppin, Jr. It can be accessed through the American Physiological Society website under the Peer Review Resources section.

CRITICISM OF PEER REVIEW

A major criticism of peer review is that there is little evidence that the process actually works, that it is actually an effective screen for good quality scientific work, and that it actually improves the quality of scientific literature. As a 2002 study published in the Journal of the American Medical Association concluded, ‘Editorial peer review, although widely used, is largely untested and its effects are uncertain’ ( 25 ). Critics also argue that peer review is not effective at detecting errors. Highlighting this point, an experiment by Godlee et al. published in the British Medical Journal (BMJ) inserted eight deliberate errors into a paper that was nearly ready for publication, and then sent the paper to 420 potential reviewers ( 7 ). Of the 420 reviewers that received the paper, 221 (53%) responded, the average number of errors spotted by reviewers was two, no reviewer spotted more than five errors, and 35 reviewers (16%) did not spot any.

Another criticism of peer review is that the process is not conducted thoroughly by scientific conferences with the goal of obtaining large numbers of submitted papers. Such conferences often accept any paper sent in, regardless of its credibility or the prevalence of errors, because the more papers they accept, the more money they can make from author registration fees ( 26 ). This misconduct was exposed in 2014 by three MIT graduate students by the names of Jeremy Stribling, Dan Aguayo and Maxwell Krohn, who developed a simple computer program called SCIgen that generates nonsense papers and presents them as scientific papers ( 26 ). Subsequently, a nonsense SCIgen paper submitted to a conference was promptly accepted. Nature recently reported that French researcher Cyril Labbé discovered that sixteen SCIgen nonsense papers had been used by the German academic publisher Springer ( 26 ). Over 100 nonsense papers generated by SCIgen were published by the US Institute of Electrical and Electronic Engineers (IEEE) ( 26 ). Both organisations have been working to remove the papers. Labbé developed a program to detect SCIgen papers and has made it freely available to ensure publishers and conference organizers do not accept nonsense work in the future. It is available at this link: http://scigendetect.on.imag.fr/main.php ( 26 ).

Additionally, peer review is often criticized for being unable to accurately detect plagiarism. However, many believe that detecting plagiarism cannot practically be included as a component of peer review. As explained by Alice Tuff, development manager at Sense About Science, ‘The vast majority of authors and reviewers think peer review should detect plagiarism (81%) but only a minority (38%) think it is capable. The academic time involved in detecting plagiarism through peer review would cause the system to grind to a halt’ ( 27 ). Publishing house Elsevier began developing electronic plagiarism tools with the help of journal editors in 2009 to help improve this issue ( 27 ).

It has also been argued that peer review has lowered research quality by limiting creativity amongst researchers. Proponents of this view claim that peer review has repressed scientists from pursuing innovative research ideas and bold research questions that have the potential to make major advances and paradigm shifts in the field, as they believe that this work will likely be rejected by their peers upon review ( 28 ). Indeed, in some cases peer review may result in rejection of innovative research, as some studies may not seem particularly strong initially, yet may be capable of yielding very interesting and useful developments when examined under different circumstances, or in the light of new information ( 28 ). Scientists that do not believe in peer review argue that the process stifles the development of ingenious ideas, and thus the release of fresh knowledge and new developments into the scientific community.

Another issue that peer review is criticized for, is that there are a limited number of people that are competent to conduct peer review compared to the vast number of papers that need reviewing. An enormous number of papers published (1.3 million papers in 23,750 journals in 2006), but the number of competent peer reviewers available could not have reviewed them all ( 29 ). Thus, people who lack the required expertise to analyze the quality of a research paper are conducting reviews, and weak papers are being accepted as a result. It is now possible to publish any paper in an obscure journal that claims to be peer-reviewed, though the paper or journal itself could be substandard ( 29 ). On a similar note, the US National Library of Medicine indexes 39 journals that specialize in alternative medicine, and though they all identify themselves as “peer-reviewed”, they rarely publish any high quality research ( 29 ). This highlights the fact that peer review of more controversial or specialized work is typically performed by people who are interested and hold similar views or opinions as the author, which can cause bias in their review. For instance, a paper on homeopathy is likely to be reviewed by fellow practicing homeopaths, and thus is likely to be accepted as credible, though other scientists may find the paper to be nonsense ( 29 ). In some cases, papers are initially published, but their credibility is challenged at a later date and they are subsequently retracted. Retraction Watch is a website dedicated to revealing papers that have been retracted after publishing, potentially due to improper peer review ( 30 ).

Additionally, despite its many positive outcomes, peer review is also criticized for being a delay to the dissemination of new knowledge into the scientific community, and as an unpaid-activity that takes scientists’ time away from activities that they would otherwise prioritize, such as research and teaching, for which they are paid ( 31 ). As described by Eva Amsen, Outreach Director for F1000Research, peer review was originally developed as a means of helping editors choose which papers to publish when journals had to limit the number of papers they could print in one issue ( 32 ). However, nowadays most journals are available online, either exclusively or in addition to print, and many journals have very limited printing runs ( 32 ). Since there are no longer page limits to journals, any good work can and should be published. Consequently, being selective for the purpose of saving space in a journal is no longer a valid excuse that peer reviewers can use to reject a paper ( 32 ). However, some reviewers have used this excuse when they have personal ulterior motives, such as getting their own research published first.

RECENT INITIATIVES TOWARDS IMPROVING PEER REVIEW

F1000Research was launched in January 2013 by Faculty of 1000 as an open access journal that immediately publishes papers (after an initial check to ensure that the paper is in fact produced by a scientist and has not been plagiarised), and then conducts transparent post-publication peer review ( 32 ). F1000Research aims to prevent delays in new science reaching the academic community that are caused by prolonged publication times ( 32 ). It also aims to make peer reviewing more fair by eliminating any anonymity, which prevents reviewers from delaying the completion of a review so they can publish their own similar work first ( 32 ). F1000Research offers completely open peer review, where everything is published, including the name of the reviewers, their review reports, and the editorial decision letters ( 32 ).

PeerJ was founded by Jason Hoyt and Peter Binfield in June 2012 as an open access, peer reviewed scholarly journal for the Biological and Medical Sciences ( 33 ). PeerJ selects articles to publish based only on scientific and methodological soundness, not on subjective determinants of ‘impact ’, ‘novelty’ or ‘interest’ ( 34 ). It works on a “lifetime publishing plan” model which charges scientists for publishing plans that give them lifetime rights to publish with PeerJ, rather than charging them per publication ( 34 ). PeerJ also encourages open peer review, and authors are given the option to post the full peer review history of their submission with their published article ( 34 ). PeerJ also offers a pre-print review service called PeerJ Pre-prints, in which paper drafts are reviewed before being sent to PeerJ to publish ( 34 ).

Rubriq is an independent peer review service designed by Shashi Mudunuri and Keith Collier to improve the peer review system ( 35 ). Rubriq is intended to decrease redundancy in the peer review process so that the time lost in redundant reviewing can be put back into research ( 35 ). According to Keith Collier, over 15 million hours are lost each year to redundant peer review, as papers get rejected from one journal and are subsequently submitted to a less prestigious journal where they are reviewed again ( 35 ). Authors often have to submit their manuscript to multiple journals, and are often rejected multiple times before they find the right match. This process could take months or even years ( 35 ). Rubriq makes peer review portable in order to help authors choose the journal that is best suited for their manuscript from the beginning, thus reducing the time before their paper is published ( 35 ). Rubriq operates under an author-pay model, in which the author pays a fee and their manuscript undergoes double-blind peer review by three expert academic reviewers using a standardized scorecard ( 35 ). The majority of the author’s fee goes towards a reviewer honorarium ( 35 ). The papers are also screened for plagiarism using iThenticate ( 35 ). Once the manuscript has been reviewed by the three experts, the most appropriate journal for submission is determined based on the topic and quality of the paper ( 35 ). The paper is returned to the author in 1-2 weeks with the Rubriq Report ( 35 ). The author can then submit their paper to the suggested journal with the Rubriq Report attached. The Rubriq Report will give the journal editors a much stronger incentive to consider the paper as it shows that three experts have recommended the paper to them ( 35 ). Rubriq also has its benefits for reviewers; the Rubriq scorecard gives structure to the peer review process, and thus makes it consistent and efficient, which decreases time and stress for the reviewer. Reviewers also receive feedback on their reviews and most significantly, they are compensated for their time ( 35 ). Journals also benefit, as they receive pre-screened papers, reducing the number of papers sent to their own reviewers, which often end up rejected ( 35 ). This can reduce reviewer fatigue, and allow only higher-quality articles to be sent to their peer reviewers ( 35 ).

According to Eva Amsen, peer review and scientific publishing are moving in a new direction, in which all papers will be posted online, and a post-publication peer review will take place that is independent of specific journal criteria and solely focused on improving paper quality ( 32 ). Journals will then choose papers that they find relevant based on the peer reviews and publish those papers as a collection ( 32 ). In this process, peer review and individual journals are uncoupled ( 32 ). In Keith Collier’s opinion, post-publication peer review is likely to become more prevalent as a complement to pre-publication peer review, but not as a replacement ( 35 ). Post-publication peer review will not serve to identify errors and fraud but will provide an additional measurement of impact ( 35 ). Collier also believes that as journals and publishers consolidate into larger systems, there will be stronger potential for “cascading” and shared peer review ( 35 ).

CONCLUDING REMARKS

Peer review has become fundamental in assisting editors in selecting credible, high quality, novel and interesting research papers to publish in scientific journals and to ensure the correction of any errors or issues present in submitted papers. Though the peer review process still has some flaws and deficiencies, a more suitable screening method for scientific papers has not yet been proposed or developed. Researchers have begun and must continue to look for means of addressing the current issues with peer review to ensure that it is a full-proof system that ensures only quality research papers are released into the scientific community.

• Systematic review
• Open access
• Published: 15 July 2024

Teamwork and implementation of innovations in healthcare and human service settings: a systematic review

• Elizabeth A. McGuier   ORCID: orcid.org/0000-0002-6219-6358 1 ,
• David J. Kolko 1 ,
• Gregory A. Aarons 2 , 3 , 4 ,
• Allison Schachter 5 , 6 ,
• Mary Lou Klem 7 ,
• Matthew A. Diabes 8 ,
• Laurie R. Weingart 8 ,
• Eduardo Salas 9 &
• Courtney Benjamin Wolk 5 , 6  

Implementation Science volume  19 , Article number:  49 ( 2024 ) Cite this article

851 Accesses

10 Altmetric

Metrics details

Implementation of new practices in team-based settings requires teams to work together to respond to new demands and changing expectations. However, team constructs and team-based implementation approaches have received little attention in the implementation science literature. This systematic review summarizes empirical research examining associations between teamwork and implementation outcomes when evidence-based practices and other innovations are implemented in healthcare and human service settings.

We searched MEDLINE, CINAHL, APA PsycINFO and ERIC for peer-reviewed empirical articles published from January 2000 to March 2022. Additional articles were identified by searches of reference lists and a cited reference search for included articles (completed in February 2023). We selected studies using quantitative, qualitative, or mixed methods to examine associations between team constructs and implementation outcomes in healthcare and human service settings. We used the Mixed Methods Appraisal Tool to assess methodological quality/risk of bias and conducted a narrative synthesis of included studies. GRADE and GRADE-CERQual were used to assess the strength of the body of evidence.

Searches identified 10,489 results. After review, 58 articles representing 55 studies were included. Relevant studies increased over time; 71% of articles were published after 2016. We were unable to generate estimates of effects for any quantitative associations because of very limited overlap in the reported associations between team variables and implementation outcomes. Qualitative findings with high confidence were: 1) Staffing shortages and turnover hinder implementation; 2) Adaptive team functioning (i.e., positive affective states, effective behavior processes, shared cognitive states) facilitates implementation and is associated with better implementation outcomes; Problems in team functioning (i.e., negative affective states, problematic behavioral processes, lack of shared cognitive states) act as barriers to implementation and are associated with poor implementation outcomes; and 3) Open, ongoing, and effective communication within teams facilitates implementation of new practices; poor communication is a barrier.

Conclusions

Teamwork matters for implementation. However, both team constructs and implementation outcomes were often poorly specified, and there was little overlap of team constructs and implementation outcomes studied in quantitative studies. Greater specificity and rigor are needed to understand how teamwork influences implementation processes and outcomes. We provide recommendations for improving the conceptualization, description, assessment, analysis, and interpretation of research on teams implementing innovations.

Trial registration

This systematic review was registered in PROSPERO, the international prospective register of systematic reviews. Registration number: CRD42020220168.

Peer Review reports

Contributions to the Literature:

This paper reviews more than 20 years of research on teams and implementation of new practices in healthcare and human service settings.

We concluded with high confidence that adaptive team functioning is associated with better implementation outcomes and problems in team functioning are associated with poorer implementation outcomes. While not surprising, the implementation science literature has lacked clear empirical evidence for this finding.

Use of the provided recommendations will improve the quality of future research on teams and implementation of evidence-based practices.

Healthcare and human service providers (e.g., clinicians, case managers) often work in team-based settings where professionals work collaboratively with one another and service recipients toward shared goals [ 1 , 2 ]. Team-based care is intended to include multiple professionals with varying skills and expertise [ 1 , 3 ]. It requires shared responsibility for outcomes and increases team members’ dependence on one another to complete work [ 1 , 3 , 4 ]. Effective team-based care and higher quality teamwork are associated with improvements in care access and quality, patient safety, patient satisfaction, clinical outcomes, and costs [ 2 , 4 , 5 , 6 , 7 , 8 , 9 ].

We use the term ‘teamwork’ to refer to an array of team constructs using the input-mediator-outcome-input (IMOI) framework (Fig.  1 ) [ 10 , 11 , 12 ]. The IMOI framework recognizes that team interactions are dynamic and complex, with processes unfolding over time and feedback loops between processes, outcomes, and inputs [ 10 ]. Team inputs include team structure and composition, task demands, and contextual features [ 13 ]. Mediators are aspects of team functioning (i.e., what team members think, feel, and do [ 12 ]) through which inputs influence outcomes. These processes and emergent states may be cognitive, affective, or behavioral [ 5 , 14 , 15 , 16 ]. Team effectiveness outcomes are multidimensional and include team performance as well as team viability and the impact of the team on members’ development [ 12 , 17 , 18 , 19 ].

Conceptual model of team effectiveness and key terminology. Figure adapted from “Advancing research on teams and team effectiveness in implementation science: An application of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework” by E.A. McGuier, D.J. Kolko, N.A. Stadnick, L. Brookman-Frazee, C.B. Wolk, C.T. Yuan, C.S. Burke, & G.A. Aarons, 2023, Implementation Research and Practice , 4 , 26334895231190855. [CC BY-NC]

Implementation of new practices in team-based service settings requires team members to work together to respond to changing demands and expectations. Extensive research has identified barriers and facilitators to implementation of new practices at the individual provider, organization, and system levels; however, the team level has received little empirical attention [ 20 , 21 ]. This is a problem because implementation efforts increasingly rely on teams, and responses to a new practice are likely to be influenced by team characteristics and processes. See McGuier and colleagues [ 20 ] for an overview of team constructs in the context of implementation science and the Exploration, Preparation, Implementation, Sustainment (EPIS) framework [ 22 , 23 ]. Given increasing use of team-based care and interest in implementation strategies targeting teams, examining how teamwork is associated with implementation processes and outcomes is critical. This systematic review identified and summarized empirical research examining associations between teamwork and implementation outcomes when evidence-based practices (EBPs) and other innovations were implemented in healthcare and human service settings.

This systematic review was registered (PROSPERO; registration number: CRD42020220168) and conducted following the published protocol [ 24 ]. The review was conducted in accordance with PRISMA and SWiM guidance [ 25 , 26 ]; relevant checklists are in Additional File 1.

Information sources and search strategy

We searched the following databases: MEDLINE (Ovid), CINAHL (Ebsco), APA PsycINFO (Ovid), and ERIC (Ebsco). Database searches were run on August 7, 2020, and again on March 8, 2022. For all searches, a publication date from 2000 to current was applied; there were no language restrictions (see [ 24 ]). An experienced health sciences librarian (MLK) designed the Ovid MEDLINE search and translated that search for use in the other databases (see additional file in [ 24 ]). The search strings consisted of controlled vocabulary (when available) and natural language terms representing concepts of teamwork and implementation science or innovation or evidence-based practice. Results were downloaded to an EndNote (version X9.3.3) library and duplicate records removed [ 27 ]. Additional relevant articles were identified by hand searches of reference lists of included articles, a cited reference search for included articles in the Web of Science (Clarivate) bibliographic database (completed in February 2023), and requests sent to implementation science listservs and centers for suggestions of relevant articles.

Eligibility criteria

We included empirical journal articles describing studies using quantitative, qualitative, or mixed methods. Study protocols, reviews, and commentaries were excluded. All studies were conducted in healthcare or human service settings (e.g., hospitals, clinics, child welfare) and described the implementation of a practice to improve patient care. Studies of interventions to improve teamwork (e.g., team building interventions) and studies of teams created to implement the innovation (e.g., quality improvement teams, implementation support teams) were excluded. Eligible studies assessed at least one team construct and described its influence on implementation processes and outcomes.

Changes from protocol

Several changes were made from our systematic review protocol (PROSPERO CRD42020220168; [ 24 ]). Specifically, during the full-text review stage, we broadened the scope from team functioning (i.e., processes and states) to include team structure and performance because of the small number of studies that assessed and reported specific processes or states. This change increased the number of included studies. Similarly, because implementation outcomes were often inconsistently defined and poorly reported [ 28 , 29 , 30 ], we broadened our scope to include studies that identified team constructs as implementation determinants (i.e., barriers/facilitators) without explicitly defining and measuring an implementation outcome. Because of changes in university access to bibliographic databases, the cited reference search was performed in the Web of Science only instead of the Web of Science and Scopus. This bibliographic database indexes more than 21,000 scientific journals [ 31 ]. Lastly, because of time and resource constraints, we did not search conference abstracts or contact authors for unreported data.

Selection process and data extraction

Title/abstract screening and review of full-text articles were conducted by pairs of trained independent reviewers in DistillerSR. Conflicts were resolved through re-review, discussion between reviewers, and when needed, discussion with a senior team member (EAM). A final review of all included articles was conducted by EAM. Relevant data from each article was extracted into an Excel spreadsheet by one reviewer (AS). A second reviewer (EAM) conducted a line-by-line review and verification. Our data extraction form was informed by existing forms and guides (e.g., [ 32 , 33 ]). For each included study, we extracted information on measures of teamwork and implementation-relevant outcomes, characteristics of the setting, teams, and participants, analysis methods, and results. For quantitative studies, we recorded correlation coefficients and/or regression coefficients as standardized metrics of association. For qualitative studies, we recorded themes [ 33 ].

Quality and risk of bias assessment

The Mixed Methods Appraisal Tool (MMAT) [ 34 ] was used to evaluate quality and risk of bias for each included study. Multiple publications from the same study were evaluated separately because they reported different outcomes. Consistent with Powell and colleagues [ 35 ], quality evaluations were only made for the components of the study relevant to our question. Quality evaluations were conducted by two independent reviewers (EAM, MAD) with discrepancies resolved through consensus discussion. After completing the MMAT, the reviewers jointly categorized each article as high, moderate, or low quality. High quality studies were those with affirmative responses to all MMAT questions. Moderate quality studies had at least one minor methodological problem, and low-quality studies had serious flaws (e.g., qualitative studies with poor coherence between data, analysis, and conclusions; quantitative studies with biased samples and/or inappropriate statistical analyses).

We rated the relevance of each publication to our research question as high, moderate, or low. Highly relevant studies reported implementation of a well-defined innovation, thoroughly described team constructs and implementation outcomes, and clearly linked team constructs to implementation outcomes. Most studies rated as low relevance provided very limited information about teamwork and/or implementation outcomes. Studies that only described barriers/facilitators were rated as low or moderate relevance. Ratings were conducted by two independent reviewers (EAM, CBW) with discrepancies resolved through consensus discussion.

Data synthesis

We conducted a narrative synthesis of included studies following guidelines for synthesis without meta-analysis (SWiM) [ 36 ]. We prioritized reporting of high quality, highly relevant studies. Studies categorized as low quality and/or low relevance were not included in the synthesis but are included in the description of study characteristics to convey the breadth of the literature. We organized studies based on the IMOI framework (i.e., team inputs, processes/states, and outputs) and organized studies of processes/states by affective, behavioral, and cognitive constructs when possible. Because of the heterogeneity in team constructs and implementation outcomes, we were unable to quantitatively synthesize results using meta-analysis or formally investigate heterogeneity; this challenge is common in implementation science systematic reviews [ 30 ]. We assessed the strength of the overall body of evidence with GRADE for quantitative studies [ 37 ] and GRADE-CERQual for qualitative studies [ 38 , 39 ]. GRADE results in ratings of high, moderate, low, or very low quality of evidence for each outcome of interest. GRADE-CERQual results in ratings of high, moderate, low, or very low confidence in each review finding. GRADE ratings were made independently with discrepancies resolved through consensus discussion; GRADE-CERQual ratings were made through iterative discussions as recommended [ 39 ]. All ratings and decisions were made by the first and senior authors.

Search results

Our initial search, after removal of duplicates, yielded 7181 results. The second search (August 2020-March 2022) captured an additional 1341 results. The cited reference search yielded 1961 results. A total of 10,489 results were included in title/abstract review. Figure  2 provides a PRISMA flow diagram for included studies. After full-text review, 58 articles from 55 studies were included in analyses [ 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 ].

PRISMA flow diagram of included articles. From: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. https://doi.org/10.1136/bmj.n71 . For more information, visit: http://www.prisma-statement.org/

As shown in Fig.  3 , publications on teamwork and implementation have increased substantially since 2000. Three articles on this topic (5%) were published between 2000 and 2007, 14 (24%) between 2008 and 2015, and 41 (71%) between 2016 and early 2023.

Included articles by year of publication

Study characteristics

Interrater agreement was good for assessment of study quality (81% agreement on MMAT questions) and ratings of relevance (88% agreement). There were 20 high quality articles, 23 moderate quality articles, and 15 low quality articles. Fourteen articles were rated as high relevance, 22 as moderate, and 22 as low relevance. Only 4 were rated as both high quality and high relevance. We report study characteristics for all 58 eligible articles. Our narrative synthesis includes 32 articles categorized as moderate/high quality and moderate/high relevance; it excludes 26 articles categorized as low quality and/or relevance.

Studies were conducted in inpatient healthcare ( n  = 22), outpatient/ambulatory healthcare ( n  = 21), mental health settings ( n  = 9), and other settings (e.g., residential facilities, multiple settings; n  = 6). There were 33 qualitative, 15 quantitative, and 10 mixed methods studies. All quantitative studies were descriptive observational studies.

Most studies examined team processes/states ( n  = 53); fewer examined team inputs ( n  = 27). Only two studies examined a team effectiveness outcome. The most common implementation outcomes were fidelity ( n  = 16) and other specified implementation outcomes (e.g., “extent of use,” “implementation success”) ( n  = 15). Less frequently identified implementation outcomes included adoption ( n  = 5), sustainment ( n  = 4), reach ( n  = 4), and perceptions of the innovation (e.g., acceptability, appropriateness, feasibility; n  = 3). Approximately one-third of studies ( n  = 21) did not report specific implementation outcomes but described implementation determinants (i.e., barriers and facilitators).

Synthesis: team inputs & implementation outcomes

Team inputs examined in studies included team stability/instability and staffing shortages, aspects of team structure and composition, interdependence, and hierarchy and professional roles. Quantitative findings are presented in Table  1 . A CERQual Summary of Qualitative Findings related to team inputs is shown in Table  2 . A CERQual Evidence Profile is provided in Additional File 2 (Table A1).

Team stability/instability and staffing shortages

Team stability/instability (i.e., consistency in membership over time) was examined in one mixed methods study [ 48 , 49 ] and three qualitative studies [ 70 , 81 , 94 ]. A study of surgical teams found variations in membership stability but no association between stability and “implementation success” (i.e., composite measure based on number of uses of new technique, proportion of uses, and changes in use) [ 48 , 49 ]. The authors suggested that stability facilitates the development of team coordination but that selecting small and exclusive teams may limit the spread of innovations within the organization. Another study found that a dedicated and stable team in which members were selected and trained together in the use of a new surgical technique led to quicker uptake and better integration into practice, theorizing that dedicated and stable teams increased trust, motivation, and collaborative problem-solving [ 81 ]. However, dedicated teams were difficult to sustain, and some sites instead used rotating team members from a larger pool of trained staff. In rural primary care, stability of team members facilitated sustainment of memory care clinics [ 70 ]. Lastly, another study in primary care found mixed perceptions of stable vs. rotating staff when adding a new team role (i.e., health coach); some team members liked rotating through different roles while others wanted more stability [ 94 ]. Across studies, we found that dedicated and stable team members facilitate implementation while instability in team membership is a barrier to implementation (moderate confidence).

Qualitative studies identified staffing shortages and turnover on teams as barriers to implementation [ 50 , 67 , 75 , 78 , 92 ]. In Veterans Health Administration (VA) clinics, “inadequate staffing posed an insurmountable barrier,” hindering communication and delivery of optimal care during the implementation of the patient-centered medical home (PCMH) model [ 92 ]. Similarly, staff shortages, turnover, and high workloads hindered guideline implementation in Kenyan hospitals [ 75 ]. Two studies found negative impacts of staffing shortages and turnover on sustainment. Staff turnover contributed to discontinuity in Dialectical Behavior Therapy (DBT) team members [ 78 ], and appropriate staffing (i.e., ensuring manageable workloads) and blocking time for team members were identified as critical to sustainment of a team-based model in the VA [ 67 ]. We found that staffing shortages and turnover hinder implementation (high confidence).

Team structure/composition

Studies examined multiple aspects of team structure and composition, specifically team size, workload, longevity (i.e., how long team members had worked together), history of change, and team member characteristics. Team size was examined in two studies of DBT. In a mixed methods study, team size was positively correlated with fidelity, and qualitative data suggested that team size may increase as a result of successful implementation [ 47 ]. In contrast, another study found that DBT team size was not associated with the number of DBT components adopted and was negatively associated with reach, suggesting reach may reflect high workloads [ 72 ]. In VA mental health clinics, team workload (i.e., number of patients seen) was negatively associated with sustainment of trauma-focused therapies [ 68 ]. In these studies, team longevity and history of change were not associated with implementation outcomes [ 47 , 68 ]. Team member characteristics, specifically team member competency/expertise, experience, and commitment/engagement, were identified as facilitators of implementation in some qualitative studies [ 40 , 70 , 81 , 84 , 95 ].

Overall, few findings could be made from quantitative studies examining team structure and composition. Two studies of team size found mixed results, and workload, longevity, and history of change were examined in only one study each. Across qualitative studies, we found team member competency/expertise, experience, and commitment/engagement facilitate implementation (moderate confidence).

Team interdependence

One quantitative study examined team interdependence [ 65 ]. In multidisciplinary child abuse teams implementing a mental health screening/referral protocol, task interdependence (i.e., reliance on team members to share resources and coordinate workflows) was positively associated with reach but not time to adoption. Outcome interdependence (i.e., extent to which outcomes are evaluated at the team vs. individual level) was significantly negatively correlated with time to adoption but not reach. Neither task nor outcome interdependence were associated with team members’ perceptions of acceptability, appropriateness, or feasibility of the innovation [ 65 ]. Because only one study examined interdependence, no review findings were made.

Hierarchy & professional roles

Hierarchy, power distributions, and rigid roles were identified as barriers to implementation in several qualitative studies [ 50 , 53 , 74 , 97 ]. Flatter hierarchies (i.e., more equal distribution of power and authority) supported guideline implementation in pediatric primary care; practices with low compliance to guidelines had more hierarchical relationships while practices with high compliance had more shared decision-making [ 97 ]. In a setting with hierarchy and rigid division of roles, nurses trained in an innovation reported concern that their decisions would be questioned by physicians without expertise in the innovation but more authority [ 74 ]. Similarly, in surgical teams, rigid professional roles and a hierarchical team culture constrained open discussion and created contention over how and when a “time-out” should be completed, resulting in inconsistent use and poor fidelity [ 50 , 53 ]. Across studies, we found that in multidisciplinary settings, rigid professional roles, hierarchical relationships, and power differentials are barriers to implementation (moderate confidence).

Summary of team inputs & implementation outcomes

There was no overlap among team input variables and implementation outcomes examined in quantitative studies (Table  1 ). Accordingly, we were unable to generate estimates of effects or ratings of evidence quality. Qualitative review findings are shown in Table  2 . We found: 1) Dedicated and stable team members facilitate implementation while instability in team membership is a barrier to implementation (moderate confidence); 2) Staffing shortages and turnover hinder implementation (high confidence); 3) Team member competency/expertise, experience, and commitment/engagement facilitate implementation (moderate confidence); and 4) In multidisciplinary settings, rigid professional roles, hierarchical relationships, and power differentials are barriers to implementation (moderate confidence).

Synthesis: team processes/states & implementation outcomes

Studies examined overall team functioning as well as specific affective states, behavioral processes, and cognitive states. Quantitative findings are presented in Table  3 , and a GRADE Evidence Profile is provided in Additional File 2 (Table A2). A CERQual Summary of Qualitative Findings related to team processes and states is shown in Table  4 . A CERQual Evidence Profile is provided in Additional File 2 (Table A3).

Overall team functioning

Nine studies examined quantitative associations between overall team functioning and implementation outcomes. Team functioning was positively associated with intervention fidelity in 2 of 3 studies. One study examined implementation of transition programs for adolescents with chronic health conditions in 29 teams. More positive team climate, measured by the Team Climate Inventory (i.e., shared vision, participative safety, task orientation, support for innovation), at study start was associated with greater improvements in quality of chronic care delivery one year later [ 45 ]. Additionally, improvements in team climate were associated with greater improvement in care delivery [ 45 ]. These findings were consistent across teams working with different patient populations, suggesting the influence of team climate generalizes across teams and settings [ 45 ]. Greater team climate for innovation was also associated with greater fidelity (i.e., implementation of more program elements) among DBT teams [ 47 ]. In contrast, no significant associations were found between team climate and fidelity to a multifaceted cardiovascular disease management intervention, with qualitative data suggesting variation in the influence of teamwork across practices [ 77 ]. There was no overlap in the metrics of association reported in these studies; therefore, we were unable to generate an estimate of the effect of team functioning on fidelity. The quality of the evidence for fidelity was rated very low because of serious methodological limitations, serious inconsistency, and very serious imprecision due to the small number of studies.

Three studies examined associations between teamwork and adoption, with no significant associations found. The first study found that teamwork climate (i.e., perceived quality of collaboration between personnel) was not significantly associated with adoption of a comprehensive safety program in intensive care units, although there were associations between adoption and organizational constructs (e.g., lower safety climate, more management support) [ 59 ]. In a study of DBT teams, neither positive nor negative team functioning was associated with the number of DBT modes adopted [ 72 ]. The third study assessed relational coordination (i.e., shared goals, communication, respect) in primary care practices implementing patient engagement strategies. Relational coordination was high across practices initially and did not differ for practices with high vs. low adoption, although it increased over time in practices with high adoption [ 83 ]. There was no overlap in the metrics of association reported in these studies; therefore, we were unable to generate an estimate of the effect of team functioning on adoption. The quality of the evidence was rated very low because of serious methodological limitations and very serious imprecision due to the small number of studies.

Reach and sustainment were each examined in one quantitative study. DBT teams with more negative functioning had greater reach, suggesting that reach may reflect high workloads; positive functioning was not associated with reach [ 72 ]. In VA mental health clinics, team functioning was positively correlated with sustainment of evidence-based trauma-focused psychotherapies and significantly associated with sustainment after controlling for covariates [ 68 ]. Two studies examined other implementation outcomes. One found that better team functioning was associated with greater implementation of changes to improve access to care in VA clinics [ 62 ]. In the other, primary care practices reporting better teamwork were more likely to be in later stages of transformation to PCMHs than practices with poorer teamwork [ 88 ]. Because of the small number of studies examining reach, sustainment, and other implementation outcomes, we were unable to generate estimates of effects or ratings of evidence quality for these outcomes.

Our qualitative review findings are based on 12 studies describing how team functioning influenced implementation processes and outcomes. There was considerable variation across studies in how team functioning was defined and what implementation outcomes were examined. Most findings were based on relatively thin and superficial data. Studies occurred in a variety of healthcare settings with varying resources and implemented diverse interventions. We found with high confidence that 1) Adaptive team functioning, characterized by positive affective states (e.g., trust, mutual respect, belonging), effective behavior processes (e.g., frequent communication and coordination), and shared cognitive states (e.g., clear roles, shared mental models of how to provide care), facilitates implementation and is associated with better implementation outcomes; and 2) Problems in team functioning, including negative affective states (e.g., tension, lack of trust), problematic behavioral processes (e.g., conflict, competition, poor communication), and a lack of shared cognitive states (e.g., unclear roles, lack of shared awareness, competing goals), act as barriers to implementation and are associated with poor implementation outcomes.

Affective states

Specific affective states were examined in one quantitative study, three mixed methods studies, and one qualitative study. There was no overlap in the associations between affective states and implementation outcomes reported in quantitative studies (Table  3 ). In a study of multidisciplinary teams responding to child abuse, affective integration (i.e., liking, trust, respect) was positively associated with acceptability, appropriateness, and feasibility but not time to adoption or reach [ 65 ]. In DBT teams, cohesion was associated with fidelity, and qualitative data indicated that liking one’s team members and having a shared team identity were critical to effective implementation [ 47 ]. Another study of DBT teams described conflicts and lack of safety and trust within teams resulting in their dissolution [ 78 ].

Edmondson and colleagues found that psychological safety and ease of speaking up (i.e., interpersonal climate that allows members to share questions and concerns) were associated with implementation success [ 48 , 49 ]. In teams with low psychological safety, lower-status team members were hesitant to speak up, hindering change and proficiency in the new practice [ 49 ]. Psychological safety was closely related to learning behavior within the team, including speaking up with questions and concerns [ 48 , 49 ]. From the mixed methods and qualitative studies, we found that trust, cohesion, and psychological safety within teams facilitate implementation by contributing to team members’ willingness to speak up and share experiences and feedback. Negative affective states, fear of judgment, conflict, and lack of safety hinder implementation (moderate confidence).

Behavioral processes

Specific behavioral processes, including communication, learning behavior, and coordination, were examined in two quantitative studies, two mixed methods studies, and five qualitative studies. There was no overlap in the associations between behavioral processes and implementation outcomes reported in quantitative studies (Table  3 ).

Only one study reported quantitative findings for communication. Communication in DBT teams was positively associated with fidelity [ 47 ]. Qualitative studies frequently identified communication as a determinant of implementation (Table  4 ). From qualitative studies, we found that open, ongoing, and effective communication within teams facilitates implementation of new practices; poor communication is a barrier (high confidence).

Quantitative associations between team learning behavior and implementation outcomes were reported in three studies. Team learning behavior in child abuse teams was positively correlated with acceptability and feasibility; it was not associated with appropriateness, time to adoption, or reach [ 65 ]. Learning behavior was positively associated with knowledge and use of innovations in nursing teams [ 91 ] and with implementation success in surgical teams [ 48 ]. Because each of these studies examined different implementation outcomes, we were unable to generate an estimate of the effect of learning behavior or rate evidence quality.

Inter-team communication, specifically speaking up and learning from other teams (i.e., boundary spanning), was identified as a critical part of team learning processes associated with successful implementation [ 48 ]. Communication beyond the team was also identified as a facilitator of implementation in two qualitative studies [ 47 , 75 ]. We found that communication beyond the team facilitates implementation by providing opportunities for team learning (low confidence).

Lastly, two qualitative studies examined coordination among healthcare teams [ 40 , 95 ]. Findings were somewhat ambiguous and based on thin data. We found with low confidence that poor coordination among healthcare professionals interferes with providing high-quality care and can be a barrier to implementation of new approaches (low confidence).

Cognitive states

Specific cognitive states were examined in two quantitative studies. There was no overlap in the associations between cognitive states and implementation outcomes reported (Table  3 ). The first study found no significant associations between shared goals and implementation outcomes [ 65 ]. The second study found that greater team knowledge and skills were associated with greater implementation of key changes to improve access to care; team problem recognition was not associated with implementation [ 62 ].

Two studies reported qualitative findings related to shared goals. In VA mental health teams, shared mission differentiated teams with sustained high reach of EBPs from those with low reach [ 84 ]. Commitment to a shared goal consistent with the EBP supported sustainment [ 84 ]. Similarly, shared goals and vision were identified as a facilitator of DBT programs [ 47 ]. We found that shared goals, mission, and vision within teams facilitate implementation and sustainment (low confidence).

Summary of team processes/states & implementation outcomes

There was very little overlap in the reported associations between team processes/states and implementation outcomes (Table  3 ). We were unable to generate estimates of effects for any associations. When there was sufficient overlap to rate evidence quality, the evidence was rated very low quality (Table A2, Additional File 2).

Qualitative review findings are shown in Table  4 . We found the following: 1) Adaptive team functioning, characterized by positive affective states (e.g., trust, mutual respect, belonging), effective behavior processes (e.g., frequent communication and coordination), and shared cognitive states (e.g., clear roles, shared mental models of how to provide care), facilitates implementation and is associated with better implementation outcomes (high confidence); 2) Problems in team functioning, including negative affective states (e.g., tension, lack of trust), problematic behavioral processes (e.g., conflict, competition, poor communication), and a lack of shared cognitive states (e.g., unclear roles, lack of shared awareness, competing goals), act as barriers to implementation and are associated with poor implementation outcomes (high confidence); 3) Trust, cohesion, and psychological safety within teams facilitate implementation by contributing to team members’ willingness to speak up and openly share experiences and feedback. Negative affective states, fear of judgment, conflict, and lack of safety hinder implementation (moderate confidence); 4) Open, ongoing, and effective communication within teams facilitates implementation of new practices; poor communication is a barrier (high confidence); 5) Communication beyond the team facilitates implementation by providing opportunities for team learning (low confidence); 6) Poor coordination among healthcare professionals interferes with providing high-quality care and can be a barrier to implementation of new approaches (low confidence); and 7) Shared goals, mission, and vision within teams facilitate implementation and sustainment (low confidence).

Synthesis: team effectiveness outcomes & implementation outcomes

Team effectiveness outcomes are multidimensional and include performance (i.e., productivity, efficiency, and quality of the team’s work), team viability, and the impact of the team on members’ development [ 12 , 17 , 18 , 19 ]. Only two studies examined associations between team effectiveness and implementation outcomes. Quantitative findings are presented in Table  5 . One quantitative study found that team members’ ratings of team performance were associated with innovation acceptability, appropriateness, and feasibility; performance was not associated with time to adoption or reach [ 65 ]. One qualitative study found that positive outcomes for team members (e.g., increased comfort working together, greater knowledge) were associated with sustainment [ 70 ]. No studies examined associations of team viability and implementation outcomes.

Summary of team effectiveness outcomes & implementation outcomes

Only one quantitative study examined associations between a dimension of team effectiveness and implementation outcomes (Table  5 ). Accordingly, we were unable to generate ratings of evidence quality or estimates of any effects. Similarly, because there was only one qualitative study, we were unable to make a review finding.

This systematic review summarizes over 20 years of empirical literature on the associations between teamwork and implementation outcomes in the context of implementation of new practices in health and human services. Consistent with increased attention to teams and reliance on team-based models of care, as well as the growth of implementation science, studies increased substantially over time. We included studies that used quantitative, qualitative, or mixed methods, yielding a total of 58 articles representing 55 studies. Included studies spanned naturalistic implementation evaluations and planned implementation research.

Key findings with high confidence were: 1) Staffing shortages and turnover hinder implementation, 2) Adaptive team functioning, characterized by positive affective states, effective behavior processes, and shared cognitive states, facilitates implementation and is associated with better implementation outcomes. Problems in team functioning, including negative affective states, problematic behavioral processes, and a lack of shared cognitive states, act as barriers to implementation and are associated with poor implementation outcomes, and 3) Open, ongoing, and effective communication within teams facilitates implementation of new practices; poor communication is a barrier. Our results generally align with conventional wisdom and scientific research outside of healthcare, increasing confidence in the findings. Team effectiveness and change management research in other types of organizations and settings (e.g., military, aviation, space exploration) [ 98 , 99 , 100 , 101 , 102 , 103 ] is largely converging.

Overall, the literature was heterogeneous, and many studies lacked specificity regarding team composition and implementation activities and outcomes. Teamwork was defined and measured inconsistently and with limited precision across studies, which hindered our ability to draw conclusions about how teams influence implementation processes and outcomes. There was also poor measurement and reporting of implementation outcomes in many studies, consistent with a recent review of research on implementation outcomes [ 28 , 29 ]. Many studies used broad measures encompassing multiple dimensions of teamwork. Among studies that assessed specific team processes and states, there was very little overlap across constructs assessed. Qualitative studies identified a rich array of specific team processes and states; research to confirm the presence of these factors in other settings and determine their associations with implementation outcomes is needed.

In Table  6 , we summarize the limitations of existing research on teams and implementation and provide recommendations for future research. Notably, increased specificity and rigor in how teamwork is conceptualized and assessed is needed to advance our understanding of how teamwork affects implementation processes and outcomes. Limited inclusion of teams and team constructs in implementation theories, models, and frameworks has likely contributed to the neglect of teams in implementation science [ 20 ]. Updates to theories, models, and framework should consider integrating teams and team-level constructs [ 20 ]. In addition, there are well-established theories of team effectiveness that could inform hypotheses about how specific team constructs affect implementation [ 104 , 105 , 106 , 107 ].

There is considerable room for improvement in the definition and description of teams and analysis of data from teams. Describing the structure and purpose of teams, as well as interdependencies within the team, can help differentiate teams from groups of individuals who do not constitute a team, an important conceptual distinction that can be difficult to discern in study descriptions. Reporting of sampling and recruitment procedures for teams and team-level response rates is needed. For quantitative studies, use of standardized, validated measures of teamwork constructs is recommended. Researchers should be careful to base inferences about teams on team-level data. Lastly, future research should follow recommendations for improving measurement and reporting of implementation outcomes [ 29 , 108 ] and consider the multilevel context of teams in theory, measurement, analysis, and interpretation of results [ 109 ].

Limitations

As with all systematic reviews, it is possible that we failed to identify some relevant articles or data. We did not search gray literature or conference abstracts or contact authors for unreported data. Our organization of studies by the IMOI framework is likely imperfect given the broad array of team constructs included and poor reporting in many studies. We included diverse innovations intended to improve patient care, including specific EBPs, clinical practice guidelines, models of care, care bundles, procedural changes, and technological innovations. This diversity in objects of implementation reflects ongoing debates about the necessary strength of evidence for objects of implementation and varying thresholds in different contexts [ 110 ]. In this review, high quality studies tended to involve clinical interventions with strong research evidence (e.g., DBT) and clinics in structured and often team-based healthcare systems (e.g., VA). Diversity of innovations and settings provides greater external validity for our findings but may mask some findings specific to certain innovations or settings.

We only included studies of existing teams providing clinical services, however, many studies provided limited descriptions of teams, and in some cases the distinction between clinical teams and implementation/quality improvement teams was unclear. There is increasing attention to use of teams in implementation frameworks [ 20 , 111 ] and evidence that functioning of implementation teams matters [ 112 , 113 ]. Research on the composition and functioning of implementation teams is an important area for future research.

Our systematic review findings indicate that teamwork matters for implementation. However, greater specificity and rigor are needed to advance our understanding of how teamwork influences implementation processes and outcomes. We provide recommendations for improving the conceptualization, description, assessment, analysis, and interpretation of research on teams implementing new practices.

Availability of data and materials

All data cited in this review came from published papers and are therefore already available. The data created as part of the review process are included in this published article and its supplementary information files.

Mitchell P, Wynia M, Golden R, McNellis B, Okun S, Webb CE, et al. Core principles and values of effective team-based health care. NAM Perspectives. 2012. Available from: https://nam.edu/perspectives-2012-core-principles-values-of-effective-team-based-health-care/  Cited 2020 Sep 23

Reiss-Brennan B, Brunisholz KD, Dredge C, Briot P, Grazier K, Wilcox A, et al. Association of integrated team-based care with health care quality, utilization, and cost. JAMA. 2016;316(8):826–34.

Article   PubMed   Google Scholar  

Katkin JP, Kressly SJ, Edwards AR, Perrin JM, Kraft CA, Richerson JE, et al. Guiding principles for team-based pediatric care. Pediatrics. 2017;140(2):e20171489.

Lipton HL. Home is where the health is: Advancing team-based care in chronic disease management. Arch Intern Med. 2009;169(21):1945–8.

Dinh JV, Traylor AM, Kilcullen MP, Perez JA, Schweissing EJ, Venkatesh A, et al. Cross-disciplinary care: A systematic review on teamwork processes in health care. Small Group Research. 2020;51(1):125–66.

Article   Google Scholar  

Fiscella K, McDaniel SH. The complexity, diversity, and science of primary care teams. Am Psychol. 2018;73(4):451–67.

Herbert JL, Bromfield LM. A quasi-experimental study of the Multi-Agency Investigation & Support Team (MIST): A collaborative response to child sexual abuse. Child Abuse Negl. 2020;27:104827.

Google Scholar  

Reiss-Brennan B. Mental Health Integration: Normalizing Team Care. J Prim Care Community Health. 2014;5(1):55–60.

Schmutz J, Manser T. Do team processes really have an effect on clinical performance? A systematic literature review. Br J Anaesth. 2013;110(4):529–44.

Article   CAS   PubMed   Google Scholar  

Ilgen DR, Hollenbeck JR, Johnson M, Jundt D. Teams in organizations: From input-process-output models to IMOI models. Annu Rev Psychol. 2005;56:517–43.

Kozlowski SWJ, Bell BS. Work groups and teams in organization. In: Borman WC, Ilgen DR, Klimoski RJ, editors. Handbook of Psychology (Vol 12): Industrial and Organizational Psychology. New York: Wiley-Blackwell. 2003. p. 333–75. Available from: http://onlinelibrary.wiley.com/doi/abs/ https://doi.org/10.1002/9781118133880.hop212017   Cited 2020 Apr 10

Mathieu JE, Maynard MT, Rapp T, Gilson L. Team effectiveness 1997–2007: A review of recent advancements and a glimpse into the future. J Manag. 2008;34(3):410–76.

Cordery JL, Tian AW. Team Design. In: The Wiley Blackwell Handbook of the Psychology of Team Working and Collaborative Processes. John Wiley & Sons, Ltd. 2017;p. 103–28. Available from: http://onlinelibrary.wiley.com/doi/abs/ https://doi.org/10.1002/9781118909997.ch5  Cited 2021 Apr 2 

Kozlowski SWJ, Bell BS. Work groups and teams in organizations: Review update. In: Schmitt N, Highhouse S, editors. Handbook of Psychology (Vol 12): Industrial and Organizational Psychology. 2nd ed. Hoboken, NJ: Wiley; 2013. p. 111.

Kozlowski SWJ, Ilgen DR. Enhancing the effectiveness of work groups and teams. Psychol Sci Public Interest. 2006;7(3):77–124.

Marks MA, Mathieu JE, Zaccaro SJ. A temporally based framework and taxonomy of team processes. AMR. 2001;26(3):356–76.

Hackman JR. A normative model of work team effectiveness. Arlington, VA: Office of Naval Research; 1983. Available from: https://apps.dtic.mil/sti/citations/ADA136398  Cited 2022 May 16

Mathieu JE, Hollenbeck JR, van Knippenberg D, Ilgen DR. A century of work teams in the Journal of Applied Psychology. J Appl Psychol. 2017;102(3):452–67.

Rosen MA, Dietz AS. Team performance measurement. In: The Wiley Blackwell Handbook of the Psychology of Team Working and Collaborative Processes. John Wiley & Sons, Ltd; 2017. p. 479–502.

Chapter   Google Scholar  

McGuier EA, Kolko DJ, Stadnick NA, Brookman-Frazee L, Wolk CB, Yuan CT, et al. Advancing research on teams and team effectiveness in implementation science: An application of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Research and Practice. 2023;1(4):26334895231190856.

Williams NJ, Beidas RS. Annual research review: The state of implementation science in child psychology and psychiatry: A review and suggestions to advance the field. J Child Psychol Psychiatry. 2019;60:430–50.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23.

Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Sci. 2019;14(1):1.

Article   PubMed   PubMed Central   Google Scholar  

McGuier EA, Kolko DJ, Klem ML, Feldman J, Kinkler G, Diabes MA, et al. Team functioning and implementation of innovations in healthcare and human service settings: a systematic review protocol. Syst Rev. 2021;10(1):189.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339. Available from: http://www.bmj.com/content/339/bmj.b2535  Cited 2020 Apr 1

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;29(372):n71.

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 2016;104(3):240–3.

Lengnick-Hall R, Gerke DR, Proctor EK, Bunger AC, Phillips RJ, Martin JK, et al. Six practical recommendations for improved implementation outcomes reporting. Implement Sci. 2022;17(1):16.

Proctor EK, Bunger AC, Lengnick-Hall R, Gerke DR, Martin JK, Phillips RJ, et al. Ten years of implementation outcomes research: a scoping review. Implement Sci. 2023;18(1):31.

Chapman A, Rankin NM, Jongebloed H, Yoong SL, White V, Livingston PM, et al. Overcoming challenges in conducting systematic reviews in implementation science: a methods commentary. Syst Rev. 2023;12(1):116.

Pranckutė R. Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World. Publications. 2021;9(1):12.

Effective Practice and Organisation of Care (EPOC). Data collection form. EPOC Resources for review authors. Norwegian Knowledge Centre for the Health Services. 2013. Available from:  http://epoc.cochrane.org/epoc-specific-resources-review-authors .

Noyes J, Lewin S. Extracting qualitative evidence. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, et al., editors. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group; 2011. Available from: http://cqrmg.cochrane.org/supplemental-handbook-guidance

Hong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, et al. Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study. J Clin Epidemiol. 2019;1(111):49-59.e1.

Powell BJ, Patel SV, Haley AD, Haines ER, Knocke KE, Chandler S, Determinants of implementing evidence-based trauma-focused interventions for children and youth: A systematic review. Adm Policy Ment Health, et al. cited 2020 Feb 20. Available from: 2019. https://doi.org/10.1007/s10488-019-01003-3 .

Article   PubMed Central   Google Scholar  

Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;16(368):l6890.

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology. 2011;64(4):383–94.

Lewin S, Bohren M, Rashidian A, Munthe-Kaas H, Glenton C, Colvin CJ, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Implementation Sci. 2018;13(1):11–23.

Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implementation Sci. 2018;13(1):1–10.

Adjognon OL, Shin MH, Steffen MJA, Moye J, Solimeo S, Sullivan JL. Factors affecting primary care implementation for older veterans with multimorbidity in Veterans Health Administration (VA). Health Serv Res. 2021;56(S1):1057–68.

Alidina S, Chatterjee P, Zanial N, Alreja SS, Balira R, Barash D, et al. Improving surgical quality in low-income and middle-income countries: why do some health facilities perform better than others? BMJ Qual Saf. 2021;30(12):937–49.

Boltey EM, Iwashyna TJ, Hyzy RC, Watson SR, Ross C, Costa DK. Ability to predict team members’ behaviors in ICU teams is associated with routine ABCDE implementation. J Crit Care. 2019;51:192–7.

Chenoweth L, Jeon YH, Stein-Parbury J, Forbes I, Fleming R, Cook J, et al. PerCEN trial participant perspectives on the implementation and outcomes of person-centered dementia care and environments. Int Psychogeriatr. 2015;27(12):2045–57.

Costa DK, Valley TS, Miller MA, Manojlovich M, Watson SR, McLellan P, et al. ICU team composition and its association with ABCDE implementation in a quality collaborative. J Crit Care. 2018;44:1–6.

Cramm JM, Strating MMH, Nieboer AP. The role of team climate in improving the quality of chronic care delivery: a longitudinal study among professionals working with chronically ill adolescents in transitional care programmes. BMJ Open. 2014;4(5):e005369.

DePuccio MJ, Gaughan AA, Sova LN, MacEwan SR, Walker DM, Gregory ME, et al. An examination of the barriers to and facilitators of implementing nurse-driven protocols to remove indwelling urinary catheters in acute care hospitals. The Joint Commission Journal on Quality and Patient Safety. 2020;46(12):691–8.

Ditty MS, Landes SJ, Doyle A, Beidas RS. It takes a village: A mixed method analysis of inner setting variables and Dialectical Behavior Therapy implementation. Adm Policy Ment Health. 2015;42(6):672–81.

Edmondson AC. Speaking up in the operating room: How team leaders promote learning in interdisciplinary action teams. J Manage Stud. 2003;40(6):1419–52.

Edmondson AC, Bohmer RM, Pisano GP. Disrupted routines: Team learning and new technology implementation in hospitals. Adm Sci Q. 2001;46(4):685–716.

Gillespie BM, Chaboyer W, Wallis M, Fenwick C. Why isn’t “time out” being implemented? An exploratory study. Qual Saf Health Care. 2010;19(2):103–6.

Gillespie BM, Harbeck E, Lavin J, Gardiner T, Withers TK, Marshall AP. Using normalisation process theory to evaluate the implementation of a complex intervention to embed the surgical safety checklist. BMC Health Serv Res. 2018;18(1):170.

Gillespie BM, Marshall AP, Gardiner T, Lavin J, Withers TK. Impact of workflow on the use of the Surgical Safety Checklist: a qualitative study: Surgical Safety Checklist and work flow. ANZ J Surg. 2016;86(11):864–7.

Gillespie BM, Withers TK, Lavin J, Gardiner T, Marshall AP. Factors that drive team participation in surgical safety checks: a prospective study. Patient Saf Surg. 2016;10(3):1–9.

Gosling AS. Clinical team functioning and IT innovation: A study of the diffusion of a point-of-care online evidence system. J Am Med Inform Assoc. 2003;10(3):244–51.

Graetz I, Huang J, Brand R, Shortell SM, Rundall TG, Bellows J, et al. The impact of electronic health records and teamwork on diabetes care quality. American Journal of Managed Care. 2015;21(12):878–84.

PubMed   Google Scholar  

Haider S, Ali RF, Ahmed M, Humayon AA, Sajjad M, Ahmad J. Barriers to implementation of emergency obstetric and neonatal care in rural Pakistan. Sawyer A, editor. PLoS ONE. 2019;14(11):e0224161.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Harvey C, Killaspy H, Martino S, White S, Priebe S, Wright C, et al. A comparison of the implementation of Assertive Community Treatment in Melbourne, Australia and London. England Epidemiol Psychiatr Sci. 2011;20(2):151–61.

Harvey C, Killaspy H, Martino S, Johnson S. Implementation of Assertive Community Treatment in Australia: Model fidelity, patient characteristics and staff experiences. Community Ment Health J. 2012;48(5):652–61.

Hsu YJ, Marsteller JA. Who applies an intervention to influence cultural attributes in a quality improvement collaborative? J Patient Saf. 2020;16(1):1–6.

Irimu GW, Greene A, Gathara D, Kihara H, Maina C, Mbori-Ngacha D, et al. Explaining the uptake of paediatric guidelines in a Kenyan tertiary hospital – mixed methods research. BMC Health Serv Res. 2014;14(1):119.

Lall D, Engel N, Devadasan N, Horstman K, Criel B. Team-based primary health care for non-communicable diseases: complexities in South India. Health Policy Plan. 2020;35(Supplement_2):ii22-34.

Lukas CV, Mohr D, Meterko M. Team effectiveness and organizational context in the implementation of a clinical innovation. Qual Manag Health Care. 2009;18(1):25–39.

Mariani E, Vernooij-Dassen M, Koopmans R, Engels Y, Chattat R. Shared decision-making in dementia care planning: barriers and facilitators in two European countries. Aging Ment Health. 2017;21(1):31–9.

Martinez RN, Hogan TP, Balbale S, Lones K, Goldstein B, Woo C, et al. Sociotechnical perspective on implementing clinical video telehealth for veterans with spinal cord injuries and disorders. Telemedicine and e-Health. 2017;23(7):567–76.

McGuier EA, Aarons GA, Byrne KA, Campbell KA, Keeshin B, Rothenberger SD, et al. Associations between teamwork and implementation outcomes in multidisciplinary cross-sector teams implementing a mental health screening and referral protocol. Implementation Science Communications. 2023;4(1):13.

Meloncelli N, Barnett A, de Jersey S. Staff resourcing, guideline implementation and models of care for gestational diabetes mellitus management. Aust N Z J Obstet Gynaecol. 2020;60(1):115–22.

Miller CJ, Kim B, Connolly SL, Spitzer EG, Brown M, Bailey HM, et al. Sustainability of the Collaborative Chronic Care Model in outpatient mental health teams three years post-Implementation: A qualitative analysis. Adm Policy Ment Health. 2023;50(1):151–9.

Mohr DC, Rosen CS, Schnurr PP, Orazem RJ, Noorbaloochi S, Clothier BA, et al. The influence of team functioning and workload on sustainability of trauma-focused evidence-based psychotherapies. PS. 2018;69(8):879–86.

Morgan D, Kosteniuk J, O’Connell ME, Kirk A, Stewart NJ, Seitz D, et al. Barriers and facilitators to development and implementation of a rural primary health care intervention for dementia: a process evaluation. BMC Health Serv Res. 2019;19(1):709.

Morgan D, Kosteniuk J, O’Connell ME, Seitz D, Elliot V, Bayly M, et al. Factors influencing sustainability and scale-up of rural primary healthcare memory clinics: perspectives of clinic team members. BMC Health Serv Res. 2022;22(1):148.

Munce SEP, Graham ID, Salbach NM, Jaglal SB, Richards CL, Eng JJ, et al. Perspectives of health care professionals on the facilitators and barriers to the implementation of a stroke rehabilitation guidelines cluster randomized controlled trial. BMC Health Serv Res. 2017;17(1):440.

Navarro-Haro MV, Harned MS, Korslund KE, DuBose A, Chen T, Ivanoff A, et al. Predictors of adoption and reach following Dialectical Behavior Therapy Intensive Training™. Community Ment Health J. 2019;55(1):100–11.

Newbould L, Ariss S, Mountain G, Hawley MS. Exploring factors that affect the uptake and sustainability of videoconferencing for healthcare provision for older adults in care homes: a realist evaluation. BMC Med Inform Decis Mak. 2021;21(1):13.

Nyondo-Mipando AL, Woo Kinshella ML, Bohne C, Suwedi-Kapesa LC, Salimu S, Banda M, et al. Barriers and enablers of implementing bubble Continuous Positive Airway Pressure (CPAP): Perspectives of health professionals in Malawi. Ameh CA, editor. PLoS ONE. 2020;15(2):e0228915.

Nzinga J, Mbindyo P, Mbaabu L, Warira A, English M. Documenting the experiences of health workers expected to implement guidelines during an intervention study in Kenyan hospitals. Implementation Sci. 2009;4(1):44.

Papoutsi C, Poots A, Clements J, Wyrko Z, Offord N, Reed JE. Improving patient safety for older people in acute admissions: implementation of the Frailsafe checklist in 12 hospitals across the UK. Age Ageing. 2018;47(2):311–7.

Patel B, Usherwood T, Harris M, Patel A, Panaretto K, Zwar N, et al. What drives adoption of a computerised, multifaceted quality improvement intervention for cardiovascular disease management in primary healthcare settings? A mixed methods analysis using normalisation process theory. Implementation Sci. 2018;13(1):140.

Popowich AD, Mushquash AR, Pearson E, Schmidt F, Mushquash CJ. Barriers and facilitators affecting the sustainability of dialectical behaviour therapy programmes: A qualitative study of clinician perspectives. Couns Psychother Res. 2020;20(1):68–80.

Prom MC, Canelos V, Fernandez PJ, Gergen Barnett K, Gordon CM, Pace CA, et al. Implementation of integrated behavioral health care in a large medical center: Benefits, challenges, and recommendations. J Behav Health Serv Res. 2021;48(3):346–62.

Provost S, Pineault R, Grimard D, Pérez J, Fournier M, Lévesque Y, et al. Implementation of an integrated primary care cardiometabolic risk prevention and management network in Montréal: does greater coordination of care with primary care physicians have an impact on health outcomes? Health Promot Chronic Dis Prev Can. 2017;37(4):105–13.

Randell R, Honey S, Alvarado N, Greenhalgh J, Hindmarsh J, Pearman A, et al. Factors supporting and constraining the implementation of robot-assisted surgery: a realist interview study. BMJ Open. 2019;9(6):e028635.

Robbins J, McAlearney AS. Encouraging employees to speak up to prevent infections: Opportunities to leverage quality improvement and care management processes. Am J Infect Control. 2016;44(11):1224–30.

Rodriguez HP, Poon BY, Wang E, Shortell SM. Linking practice adoption of patient engagement strategies and relational coordination to patient-reported outcomes in accountable care organizations. Milbank Q. 2019;97(3):692–735.

Sayer NA, Rosen CS, Bernardy NC, Cook JM, Orazem RJ, Chard KM, et al. Context matters: Team and organizational factors associated with reach of evidence-based psychotherapies for PTSD in the Veterans Health Administration. Adm Policy Ment Health. 2017;44(6):904–18.

Shin MH, Montano ARL, Adjognon OL, Harvey KLL, Solimeo SL, Sullivan JL. Identification of implementation strategies using the CFIR-ERIC matching tool to mitigate barriers in a primary care model for older veterans. Gerontologist. 2022;63(3):439–50.

Sopcak N, Aguilar C, O’Brien MA, Nykiforuk C, Aubrey-Bassler K, Cullen R, et al. Implementation of the BETTER 2 program: a qualitative study exploring barriers and facilitators of a novel way to improve chronic disease prevention and screening in primary care. Implementation Sci. 2016;11(1):158.

Steinmo SH, Michie S, Fuller C, Stanley S, Stapleton C, Stone SP. Bridging the gap between pragmatic intervention design and theory: using behavioural science tools to modify an existing quality improvement programme to implement “Sepsis Six.” Implementation Sci. 2016;11(1):14.

Stout S, Zallman L, Arsenault L, Sayah A, Hacker K. Developing high-functioning teams: Factors associated with operating as a “real team” and implications for patient-centered medical home development. Inquiry. 2017;12(54):0046958017707296.

Svensson B, Hansson L, Markström U, Lexén A. What matters when implementing Flexible Assertive Community Treatment in a Swedish healthcare context: A two-year implementation study. Int J Ment Health. 2017;46(4):284–98.

Tadyanemhandu C, van Aswegen H, Ntsiea V. Barriers and facilitators to implementation of early mobilisation of critically ill patients in Zimbabwean and South African public sector hospitals: a qualitative study. Disabil Rehabil. 2021;44(22):6699–709.

Timmermans O, Van Linge R, Van Petegem P, Van Rompaey B, Denekens J. A contingency perspective on team learning and innovation in nursing: Team learning and innovation in nursing teams. J Adv Nurs. 2013;69(2):363–73.

Tuepker A, Kansagara D, Skaperdas E, Nicolaidis C, Joos S, Alperin M, et al. “We’ve not gotten even close to what we want to do”: A qualitative study of early patient-centered medical home implementation. J GEN INTERN MED. 2014;29(S2):614–22.

van Bruggen S, Kasteleyn MJ, Rauh SP, Meijer JS, Busch KJG, Numans ME, et al. Experiences with tailoring of primary diabetes care in well-organised general practices: a mixed-methods study. BMC Health Serv Res. 2021;21(1):1218.

Van der Wees PJ, Friedberg MW, Guzman EA, Ayanian JZ, Rodriguez HP. Comparing the implementation of team approaches for improving diabetes care in community health centers. BMC Health Serv Res. 2014;14(1):608.

von Meyenfeldt EM, van Nassau F, de Betue CTI, Barberio L, Schreurs WH, Marres GMH, et al. Implementing an enhanced recovery after thoracic surgery programme in the Netherlands: a qualitative study investigating facilitators and barriers for implementation. BMJ Open. 2022;12(1):e051513.

Wakefield BJ, Lampman MA, Paez MB, Stewart GL. Care management and care coordination within a patient-centered medical home. J Nurs Admin. 2020;50(11):565–70.

Wiener-Ogilvie S, Huby G, Pinnock H, Gillies J, Sheikh A. Practice organisational characteristics can impact on compliance with the BTS/SIGN asthma guideline: Qualitative comparative case study in primary care. BMC Fam Pract. 2008;9(1):32.

Errida A, Lotfi B. The determinants of organizational change management success: Literature review and case study. International Journal of Engineering Business Management. 2021;1(13):18479790211016270.

Phillips J, Klein JD. Change management: From theory to practice. TechTrends. 2023;67(1):189–97.

Rousseau DM, ten Have S. Evidence-based change management. Organ Dyn. 2022;51(3):100899.

Goodwin GF, Blacksmith N, Coats MR. The science of teams in the military: Contributions from over 60 years of research. Am Psychol. 2018;73(4):322–33.

DiazGranados D, Woods JJ, Wilder A, Curtis MT. Team dynamics in the air: A review of team research relevant to aviation. Human factors in aviation and aerospace. 2023. p. 199–215.

Salas E, Tannenbaum SI, Kozlowski SWJ, Miller CA, Mathieu JE, Vessey WB. Teams in space exploration: A new frontier for the science of team effectiveness. Curr Dir Psychol Sci. 2015;24(3):200–7.

Hackman JR. From causes to conditions in group research. J Organ Behav. 2012;33(3):428–44.

Tannenbaum SI, Salas E. Teams that work: The seven drivers of team effectiveness. Illustrated. New York, NY: Oxford University Press; 2020. p. 272.

Book   Google Scholar  

Tuckman BW. Developmental sequence in small groups. Psychol Bull. 1965;63:384–99.

Salas E, Stagl KC, Burke CS. 25 years of team effectiveness in organizations: Research themes and emerging needs. In: International Review of Industrial and Organizational Psychology 2004. John Wiley & Sons, Ltd. 2004; p. 47–91. Available from: https://onlinelibrary.wiley.com/doi/abs/ https://doi.org/10.1002/0470013311.ch2    Cited 2022 May 16

Proctor EK, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.

Lengnick-Hall R, Williams NJ, Ehrhart MG, Willging CE, Bunger AC, Beidas RS, et al. Eight characteristics of rigorous multilevel implementation research: a step-by-step guide. Implement Sci. 2023;18(1):52.

Wensing M, Sales A, Aarons GA, Xu D (Roman), Wilson P. Evidence for objects of implementation in healthcare: considerations for Implementation Science and Implementation Science Communications. Implementation Sci. 2022;17(1):83.

Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.

Shortell SM, Marsteller JA, Lin M, Pearson ML, Wu SY, Mendel P, et al. The role of perceived team effectiveness in improving chronic illness care. Med Care. 2004;42(11):1040–8.

Yakovchenko V, Morgan TR, Chinman MJ, Powell BJ, Gonzalez R, Park A, et al. Mapping the road to elimination: a 5-year evaluation of implementation strategies associated with hepatitis C treatment in the veterans health administration. BMC Health Serv Res. 2021;21(1):1348.

Download references

Acknowledgements

Thank you to Ikzzui Chu, Jamie Feldman, Grace Kinkler, Rachael Park, and Jaely Wright for their assistance with article screening.

This work was supported by the National Institute of Mental Health grants MH123729 (EAM), MH124914 (DJK), and MH126231 (GAA), the National Cancer Institute U01CA275118 (GAA), National Institute on Drug Abuse R01DA049891 (GAA), the Agency for Healthcare Research and Quality grant R18HS026862 (CBW), and the Collaboration and Conflict Research Lab at Carnegie Mellon University Tepper School of Business. The content of this manuscript does not represent the views of funding agencies and is solely the responsibility of the authors.

Author information

Authors and affiliations.

Department of Psychiatry, School of Medicine, University of Pittsburgh, 3811 O’Hara Street, Pittsburgh, PA, 15213, USA

Elizabeth A. McGuier & David J. Kolko

Department of Psychiatry, University of California San Diego, La Jolla, CA, USA

Gregory A. Aarons

ACTRI Dissemination and Implementation Science Center, UC San Diego, La Jolla, CA, USA

Child and Adolescent Services Research Center, San Diego, CA, USA

Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

Allison Schachter & Courtney Benjamin Wolk

Penn Implementation Science Center at the Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA, USA

Health Sciences Library System, University of Pittsburgh, Pittsburgh, PA, USA

Mary Lou Klem

Tepper School of Business, Carnegie Mellon University, Pittsburgh, PA, USA

Matthew A. Diabes & Laurie R. Weingart

Department of Psychological Sciences, Rice University, Houston, TX, USA

Eduardo Salas

You can also search for this author in PubMed   Google Scholar

Contributions

EAM conceptualized and developed the protocol for this review, participated in article screening, data extraction, quality appraisal, and analysis, and led manuscript development and writing. DJK, GAA, LRW, ES, and CBW provided guidance and input into the protocol and methods and interpretation of results. AS participated in article screening, data extraction, and preparation of supplementary files. MLK developed the search strategy and conducted database searches. MAD participated in quality appraisal. DJK, GAA, LRW, ES, and CBW reviewed multiple drafts of the manuscript and provided critical input and editing. All authors reviewed and approved the final manuscript.

Corresponding author

Correspondence to Elizabeth A. McGuier .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

GAA is a Co-Editor-in-Chief of Implementation Science and on the Editorial Board of Implementation Science Communications ; all decisions on this paper were made by another editor. The authors declare that they have no other competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

McGuier, E.A., Kolko, D.J., Aarons, G.A. et al. Teamwork and implementation of innovations in healthcare and human service settings: a systematic review. Implementation Sci 19 , 49 (2024). https://doi.org/10.1186/s13012-024-01381-9

Download citation

Received : 18 January 2024

Accepted : 07 July 2024

Published : 15 July 2024

DOI : https://doi.org/10.1186/s13012-024-01381-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation outcomes

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]