steps to conduct systematic literature review

Systematic Review

• Library Help
• What is a Systematic Review (SR)?

Steps of a Systematic Review

• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

• PICO Template
• Inclusion/Exclusion Criteria
• Database Search Log
• Review Matrix
• Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

• PROSPERO Template
• Manuscript Template
• Steps of SR (text)
• Steps of SR (visual)
• Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

• Video  with closed captions available
• Animated Storyboard

• << Previous: What is a Systematic Review (SR)?
• Next: Framing a Research Question >>
• Last Updated: Aug 26, 2024 12:37 PM
• URL: https://lib.guides.umd.edu/SR

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

• the definition of a systematic literature review
• the purpose of a systematic literature review
• the different types of systematic reviews
• how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

• Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
• Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
• Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
• Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
• Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
• Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
• Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
• Prognostic reviews examine the course of a disease and its potential outcomes.
• Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
• Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

• Decide on your team.
• Formulate your question.
• Plan your research protocol.
• Search for the literature.
• Screen the literature.
• Assess the quality of the studies.
• Extract the data.
• Analyze the results.
• Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

• Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
• Intervention : which main intervention or treatment are you considering for assessment?
• Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
• Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

• the objectives of your project
• the specific methods and processes that you’ll use
• the eligibility criteria of the individual studies
• how you plan to extract data from individual studies
• which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

• non-peer-reviewed journals
• pharmaceutical industry files
• conference proceedings
• pharmaceutical company websites
• internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

• The databases you search and which years they cover
• The dates you first run the searches, and when they’re updated
• Which strategies you use, including search terms
• The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

• Pre-screening of all titles and abstracts, and selecting those appropriate
• Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

• Were the participants really randomly allocated to their groups?
• Were the groups similar in terms of prognostic factors?
• Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

• Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
• Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
• Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

Jump to navigation

Cochrane Cochrane Interactive Learning

Cochrane interactive learning, module 1: introduction to conducting systematic reviews, about this module.

Part of the Cochrane Interactive Learning course on Conducting an Intervention Review, this module introduces you to what systematic reviews are and why they are useful. This module describes the various types and preferred format of review questions, and outlines the process of conducting systematic reviews.

 45-60 minutes

What you can expect to learn (learning outcomes).

This module will teach you to:

• Recognize features of systematic reviews as a research design
• Recognize the importance of using rigorous methods to conduct a systematic review
• Identify the types of review questions
• Identify the elements of a well-defined review question
• Understand the steps in a systematic review

Authors, contributors, and how to cite this module

Module 1 has been written and compiled by Dario Sambunjak, Miranda Cumpston and Chris Watts,  Cochrane Central Executive Team .

A full list of acknowledgements, including our expert advisors from across Cochrane, is available at the end of each module page. 

This module should be cited as: Sambunjak D, Cumpston M, Watts C. Module 1: Introduction to conducting systematic reviews. In: Cochrane Interactive Learning: Conducting an intervention review. Cochrane, 2017. Available from https://training.cochrane.org/interactivelearning/module-1-introduction-conducting-systematic-reviews .

Update and feedback

The module was last updated on September 2022.

We're pleased to hear your thoughts. If you have any questions, comments or feedback about the content of this module, please contact us .

Ohio State nav bar

The Ohio State University

• BuckeyeLink
• Find People
• Search Ohio State

Systematic Reviews

• What is a Systematic Review?

What are the Steps of a Systematic Review?

• An Overview of the Systematic Review Process
• 1. Choose the Right Kind of Review
• 2. Formulate Your Question
• 3. Establish a Team
• 4. Develop a Protocol
• 5. Conduct the Search
• 6. Select Studies
• 7. Extract Data
• 8. Synthesize Your Results
• 9. Disseminate Your Report
• Request a Librarian Consultation

Consult With a Librarian

To make an appointment to consult with an HSL librarian on your systematic review, please read our Systematic Review Policy and submit a Systematic Review Consultation Request .

To ask a question or make an appointment for assistance with a narrative review, please complete the Ask a Librarian Form .

This video from the Yale University Medical Library provides a brief overview of the process of conducting a systematic review:

Check out the rest of Yale's video series on conducting systematic searches: 

• Systematic Searches Series from Yale University
• << Previous: What is a Systematic Review?
• Next: 1. Choose the Right Kind of Review >>
• Last Updated: May 14, 2024 8:03 AM
• URL: https://hslguides.osu.edu/systematic_reviews

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved September 3, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

How to Write a Systematic Review of the Literature

Affiliations.

• 1 1 Texas Tech University, Lubbock, TX, USA.
• 2 2 University of Florida, Gainesville, FL, USA.
• PMID: 29283007
• DOI: 10.1177/1937586717747384

This article provides a step-by-step approach to conducting and reporting systematic literature reviews (SLRs) in the domain of healthcare design and discusses some of the key quality issues associated with SLRs. SLR, as the name implies, is a systematic way of collecting, critically evaluating, integrating, and presenting findings from across multiple research studies on a research question or topic of interest. SLR provides a way to assess the quality level and magnitude of existing evidence on a question or topic of interest. It offers a broader and more accurate level of understanding than a traditional literature review. A systematic review adheres to standardized methodologies/guidelines in systematic searching, filtering, reviewing, critiquing, interpreting, synthesizing, and reporting of findings from multiple publications on a topic/domain of interest. The Cochrane Collaboration is the most well-known and widely respected global organization producing SLRs within the healthcare field and a standard to follow for any researcher seeking to write a transparent and methodologically sound SLR. Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), like the Cochrane Collaboration, was created by an international network of health-based collaborators and provides the framework for SLR to ensure methodological rigor and quality. The PRISMA statement is an evidence-based guide consisting of a checklist and flowchart intended to be used as tools for authors seeking to write SLR and meta-analyses.

Keywords: evidence based design; healthcare design; systematic literature review.

PubMed Disclaimer

Similar articles

• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
• Preferred Reporting Items for Systematic Review and Meta-Analyses of individual participant data: the PRISMA-IPD Statement. Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, Tierney JF; PRISMA-IPD Development Group. Stewart LA, et al. JAMA. 2015 Apr 28;313(16):1657-65. doi: 10.1001/jama.2015.3656. JAMA. 2015. PMID: 25919529
• Systematic Reviews in Sports Medicine. DiSilvestro KJ, Tjoumakaris FP, Maltenfort MG, Spindler KP, Freedman KB. DiSilvestro KJ, et al. Am J Sports Med. 2016 Feb;44(2):533-8. doi: 10.1177/0363546515580290. Epub 2015 Apr 21. Am J Sports Med. 2016. PMID: 25899433 Review.
• The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. Liberati A, et al. J Clin Epidemiol. 2009 Oct;62(10):e1-34. doi: 10.1016/j.jclinepi.2009.06.006. Epub 2009 Jul 23. J Clin Epidemiol. 2009. PMID: 19631507
• A systematic review and meta-analysis of balance training in patients with chronic ankle instability. Guo Y, Cheng T, Yang Z, Huang Y, Li M, Wang T. Guo Y, et al. Syst Rev. 2024 Feb 12;13(1):64. doi: 10.1186/s13643-024-02455-x. Syst Rev. 2024. PMID: 38347564 Free PMC article.
• Association between infection and the onset of giant cell arteritis and polymyalgia rheumatica: a systematic review and meta-analysis. Pacoureau L, Barde F, Seror R, Nguyen Y. Pacoureau L, et al. RMD Open. 2023 Nov;9(4):e003493. doi: 10.1136/rmdopen-2023-003493. RMD Open. 2023. PMID: 37949615 Free PMC article.
• From Social Rejection to Welfare Oblivion: Health and Mental Health in Juvenile Justice in Brazil, Colombia and Spain. Carbonell Á, Georgieva S, Navarro-Pérez JJ, Botija M. Carbonell Á, et al. Int J Environ Res Public Health. 2023 May 29;20(11):5989. doi: 10.3390/ijerph20115989. Int J Environ Res Public Health. 2023. PMID: 37297594 Free PMC article. Review.
• Why is didactic transposition in disaster education needed by prospective elementary school teachers? Noviana E, Syahza A, Putra ZH, Hadriana, Yustina, Erlinda S, Putri DR, Rusandi MA, Biondi Situmorang DD. Noviana E, et al. Heliyon. 2023 Apr 18;9(4):e15413. doi: 10.1016/j.heliyon.2023.e15413. eCollection 2023 Apr. Heliyon. 2023. PMID: 37128333 Free PMC article. Review.
• Comparative analysis of efficacy of different combination therapies of α-receptor blockers and traditional Chinese medicine external therapy in the treatment of chronic prostatitis/chronic pelvic pain syndrome: Bayesian network meta-analysis. Zhang K, Zhang Y, Hong S, Cao Y, Liu C. Zhang K, et al. PLoS One. 2023 Apr 20;18(4):e0280821. doi: 10.1371/journal.pone.0280821. eCollection 2023. PLoS One. 2023. PMID: 37079509 Free PMC article.
• Search in MeSH

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.

Other Literature Sources

• scite Smart Citations

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Medical Scholarship: Systematic Reviews

• Systematic Reviews
• General Search Info
• Critical Appraisal
• Library Services
• LibKey Nomad
• Flash Talks

So you still want to try for a systematic review...

Evidence synthesis "is a type of research method that allows researchers to bring together all relevant information on a research question". ( London School of Hygiene & Tropical Medicine Centre for Evaluation )

Evidence synthesis methodology, which can produce systematic reviews, scoping reviews, rapid reviews, and other types of projects, differ from narrative reviews (which are often authored by experts in a field and are generally narrative summaries of a topic). 

Before beginning an evidence synthesis project:

• Clarify your review topic
• Decide if a systematic review is the best approach to answer the question
• Determine if you have the necessary time and resources
• Form a review team (this is not a solo activity!)
• Expect a slow and lengthy process

At this time the Hope Brings Strength Health Science Library is not equipped to assist with any systematic reviews.

CMSRU Evidence Synthesis Service: Steps in a Systematic Review  - The librarians at CMSRU are also unable to assist Rowan-Virtua SOM students with systematic reviews. 

Steps in a Systematic Review

1) determine if a review is necessary.

Before starting your review, determine if a systematic review is the best approach to answer your question. 

• What other review types did you consider?
• Will it fill a meaningful gap in existing literature?
• If you identify an existing review, assess its quality (see box 8 below).
• If a high quality review exists but was completed several years ago, a new review may be justified.
• Do you have the necessary time (12-18 months) and resources to complete a systematic review?
• Many tasks in this process must be performed by more than one person, or require specialized skills.
• at least 2 individuals who will independently screen articles and perform data extraction
• an expert in the content area of the research question
• an individual experienced in conducting systematic reviews
• a statistician, if you are also conducting a meta-analysis

2) Clarify the Question

A clearly defined, focused review begins with a well-framed question. It provides the framework for the entire review. 

• Clearly state the objectives of the review. What question are you trying to answer?
• P opulation
• I nterventions
• C omparisons
• S tudy Design

Systematic Reviews are not limited to questions about effects of interventions, they may address trends, accuracy of diagnostic tests, effectiveness of programs, etc. 

Ask yourself:

• Is the question answerable?
• Are there measurable constructs?
• Is it practical to relevant for policy/practice?

3) Define Eligibility Criteria

One of the features that distinguishes a systematic review from a narrative review is the pre-specification of criteria for including and excluding studies in the review (eligibility criteria).  Explicit criteria, based on the review’s scope and question(s), are used to include and exclude studies. 

A large number of references (study titles and abstracts) will have been found at the searching stage of the review. A proportion of these will look as though they are relevant to the review's research questions. So, having explicit criteria against which to assess studies makes the process more efficient in terms of time.

More importantly, it also helps to avoid hidden bias by having clear consistent rules about which studies are being used to answer the review's specific research questions. 

Each study needs to be compared against same criteria. To be included in the review, a study needs to meet all inclusion criteria and not meet any exclusion criteria. Inclusion/eligibility criteria include participants, interventions and comparisons and often study design. Outcomes are usually not part of the criteria, though some reviews do legitimately restrict eligibility to specific outcomes.

4) Create a Protocol

The review protocol sets out the methods to be used in the review and provides an explicit plan for your work.  Decisions about the review question, inclusion criteria, search strategy, study selection, data extraction, quality assessment, data synthesis and plans for dissemination should be addressed. If modifications to the protocol are required, these should be clearly documented and justified. Modifications may arise from a clearer understanding of the review question, and should not be made because of an awareness of the results of individual studies. 

Consider registering your protocol.  PROSPERO , from the Centre for Reviews and Dissemination at the University of York, is an international database of registered reviews in health and social care. Key features from registered reviews and recorded and maintained as a permanent record. Registration helps to promote transparency in the review process and also reduces the potential for duplication.

5) Search for Studies

A systematic search to identify studies must be comprehensive and it must strike a balance between recall and precision. In other words, don't expect to retrieve  only  relevant articles.

Most studies used in the review will be identified using electronic databases (e.g. PubMed), but identifying unpublished studies is important as well. Librarians can help to identify sources for finding grey literature.

Expect a large number of results.  Depending on the scope of your topic, your search may result in as few as a couple hundred or as many as several thousand articles.

Some key steps in searching for studies include:

• Identifying databases to be searched.
• Identifying search terminology.
• Constructing and running database searches.
• Conducting hand searches of specialized journals.
• Searching reference lists of relevant studies.
• Contacting recognized experts working in the field.
• Searching relevant grey literature sources (e.g. clinical trials registers, conference proceedings).

It’s important to note that the IOM recommends working with a librarian to plan and execute your searches. Plan to meet with a librarian to discuss your review.

6) Select Studies

A large quantity of studies needs to be assessed at this stage of the review.  This process should involve at least two members of your group to help reduce bias.  It is useful to develop an appropriate form to help select and keep track of articles that meet eligibility criteria. To be included, a study must meet all inclusion criteria and not meet any exclusion criteria. 

Employ the following process:

• Examine titles/abstracts
• Retrieve the full-text of relevant studies
• Examine the full-text to determine eligibility
• Make final decisions on study inclusion

Remember to document your decisions!  It is important to have a record of decisions made in order to ensure reproducibility and minimize errors.

This PRISMA flow diagram shows the number of studies remaining at each stage. It's a simple and useful way of documenting the study selection process and should be included in your final report.

7) Extract Data

Here, data simply refers to information about or details from a study including its methods and design, participants, setting, interventions, and results. Two researchers should independently extract data from each study in order to minimize errors and reduce the potential for bias. At a minimum, one researcher should extract data with a second person checking for accuracy.  

To standardize the process and improve the validity of the results, time and thought should be given to creating a data extraction form. These can be created using general word processing software or spreadsheets, or you can use the data extraction features in Covidence.

It is important to collect enough information during this process to sufficiently assess each study. Collecting too much or too little information may be a waste of time and result in the omission of crucial data.

8) Assess the Quality of Studies

Assessment of study quality gives an indication of the strength of the evidence provided by your review. It helps determine whether the studies are vigorous enough to guide treatment, prevention, diagnostic or policy decisions.

Quality assessment of any study is likely to consider:

• Appropriateness of study design to the research objective
• Risk of bias
• Choice of outcome measure
• Statistical issues
• Quality of reporting
• Quality of the intervention
• Generalizability

From: Centre for Reviews and Dissemination, University of York (2008)  Systematic Reviews: CRD's Guidance for Undertaking Reviews in Health Care .

9) Synthesize Data and Write the Report

Synthesis involves combining the results of the studies included in the review, summarizing their findings and drawing reliable conclusions based on the quality of the evidence. Synthesis may be done quantitatively using statistical techniques, such as a meta-analysis, or through a narrative approach.  

In general, making recommendations for practice does not fall within the purview of systematic reviews. This is typically the domain of clinical practice guidelines. Systematic review authors can make conclusions about the need for further research or draw conclusions about the usefulness of an intervention.

When preparing your report or article, refer to the  PRISMA Checklist .  Many commissioning bodies and journals have adopted PRISMA as the required methods for reporting systematic reviews.

  FAQs

 Ask a Librarian

  Subject Guides

How-To: Systematic Review

• How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses
• Five Steps to Conducting a Systematic Review

Organizational Tools

One (very important) aspect of Systematic Literature Reviews is to make them reproducible.  Your work needs to be transparent, and fully recorded, so that another person doing the research will gather the exact same results.  Here are some tools that make organizing your information easier.  Except for EndNote, the Rowan-Virtua SOM Librarians won't be able to answer technical questions on these packages.  

Grey Literature

Materials and research produced by organizations outside of the traditional commercial or academic publishing and distribution channels. Common grey literature publication types include reports (annual, research, technical, project, etc.), working papers, government documents, white papers and evaluations. Organizations that produce grey literature include government departments and agencies, civil society or non-governmental organizations, academic centers and departments, and private companies and consultants.  (Wikipedia).  Below you will find some places to look for grey literature.

• ClinicalTrials.gov
• Devices@FDA
• << Previous: Types of Reviews
• Next: Searching >>
• Last Updated: Sep 4, 2024 10:55 AM
• URL: https://rowanmed.libguides.com/medicalscholarship

• Open access
• Published: 01 August 2019

A step by step guide for conducting a systematic review and meta-analysis with simulation data

• Gehad Mohamed Tawfik 1 , 2 ,
• Kadek Agus Surya Dila 2 , 3 ,
• Muawia Yousif Fadlelmola Mohamed 2 , 4 ,
• Dao Ngoc Hien Tam 2 , 5 ,
• Nguyen Dang Kien 2 , 6 ,
• Ali Mahmoud Ahmed 2 , 7 &
• Nguyen Tien Huy 8 , 9 , 10  

Tropical Medicine and Health volume  47 , Article number:  46 ( 2019 ) Cite this article

831k Accesses

303 Citations

94 Altmetric

Metrics details

The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.

We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.

SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Introduction

The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].

Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.

Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.

Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.

Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”

Methods and results

Detailed steps for conducting any systematic review and meta-analysis.

We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.

Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].

We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.

Research question and objectives

Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.

We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.

PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.

To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.

An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)

Preliminary research and idea validation

We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.

To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).

Inclusion and exclusion criteria

Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.

For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. 2 .

PRISMA flow diagram of studies’ screening and selection

Search strategy

A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.

The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.

Search databases, import all results to a library, and exporting to an excel sheet

According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:

With all of the words: ebola virus

With at least one of the words: vaccine vaccination vaccinated immunization

Where my words occur: in the title of the article

With all of the words: EVD

Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.

Protocol writing and registration

Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.

Title and abstract screening

Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 , 10 , 11 ].

In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.

Full text downloading and screening

Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.

Manual search

One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.

We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.

We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.

Data extraction and quality assessment

This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.

Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.

In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).

Data checking

Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.

Statistical analysis

Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.

To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.

For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.

From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.

In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. 3 an example of a forest plot from the simulated analysis.

Random effect model forest plot for comparison of vaccine A versus placebo

From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.

For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.

Publication bias funnel plot for comparison of vaccine A versus placebo

Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.

Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.

Double data checking

For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.

Manuscript writing, revision, and submission to a journal

Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.

After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.

The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.

Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].

Conclusions

Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.

Availability of data and materials

Not applicable.

Abbreviations

Network meta-analysis

Principal investigator

Population, Intervention, Comparison, Outcome

Preferred Reporting Items for Systematic Review and Meta-analysis statement

Quality assessment

Sample, Phenomenon of Interest, Design, Evaluation, Research type

Systematic review and meta-analyses

Bello A, Wiebe N, Garg A, Tonelli M. Evidence-based decision-making 2: systematic reviews and meta-analysis. Methods Mol Biol (Clifton, NJ). 2015;1281:397–416.

Article   Google Scholar  

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Rys P, Wladysiuk M, Skrzekowska-Baran I, Malecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Polskie Archiwum Medycyny Wewnetrznej. 2009;119(3):148–56.

PubMed   Google Scholar  

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Gross L, Lhomme E, Pasin C, Richert L, Thiebaut R. Ebola vaccine development: systematic review of pre-clinical and clinical studies, and meta-analysis of determinants of antibody response variability after vaccination. Int J Infect Dis. 2018;74:83–96.

Article   CAS   Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, ... Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Giang HTN, Banno K, Minh LHN, Trinh LT, Loc LT, Eltobgy A, et al. Dengue hemophagocytic syndrome: a systematic review and meta-analysis on epidemiology, clinical signs, outcomes, and risk factors. Rev Med Virol. 2018;28(6):e2005.

Morra ME, Altibi AMA, Iqtadar S, Minh LHN, Elawady SS, Hallab A, et al. Definitions for warning signs and signs of severe dengue according to the WHO 2009 classification: systematic review of literature. Rev Med Virol. 2018;28(4):e1979.

Morra ME, Van Thanh L, Kamel MG, Ghazy AA, Altibi AMA, Dat LM, et al. Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis. Rev Med Virol. 2018;28(3):e1977.

Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association: JMLA. 2016;104(4):302.

Naunheim MR, Remenschneider AK, Scangas GA, Bunting GW, Deschler DG. The effect of initial tracheoesophageal voice prosthesis size on postoperative complications and voice outcomes. Ann Otol Rhinol Laryngol. 2016;125(6):478–84.

Rohatgi AJaiWa. Web Plot Digitizer. ht tp. 2014;2.

Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5(1):13.

Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.

Van Rijkom HM, Truin GJ, Van’t Hof MA. A meta-analysis of clinical studies on the caries-inhibiting effect of fluoride gel treatment. Carries Res. 1998;32(2):83–92.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Tawfik GM, Tieu TM, Ghozy S, Makram OM, Samuel P, Abdelaal A, et al. Speech efficacy, safety and factors affecting lifetime of voice prostheses in patients with laryngeal cancer: a systematic review and network meta-analysis of randomized controlled trials. J Clin Oncol. 2018;36(15_suppl):e18031-e.

Wannemuehler TJ, Lobo BC, Johnson JD, Deig CR, Ting JY, Gregory RL. Vibratory stimulus reduces in vitro biofilm formation on tracheoesophageal voice prostheses. Laryngoscope. 2016;126(12):2752–7.

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355.

RevMan The Cochrane Collaboration %J Copenhagen TNCCTCC. Review Manager (RevMan). 5.0. 2008.

Schwarzer GJRn. meta: An R package for meta-analysis. 2007;7(3):40-45.

Google Scholar  

Simms LLH. Meta-analysis versus mega-analysis: is there a difference? Oral budesonide for the maintenance of remission in Crohn’s disease: Faculty of Graduate Studies, University of Western Ontario; 1998.

Download references

Acknowledgements

This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.

Author information

Authors and affiliations.

Faculty of Medicine, Ain Shams University, Cairo, Egypt

Gehad Mohamed Tawfik

Online research Club http://www.onlineresearchclub.org/

Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien & Ali Mahmoud Ahmed

Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali, 81171, Indonesia

Kadek Agus Surya Dila

Faculty of Medicine, University of Khartoum, Khartoum, Sudan

Muawia Yousif Fadlelmola Mohamed

Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam

Dao Ngoc Hien Tam

Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam

Nguyen Dang Kien

Faculty of Medicine, Al-Azhar University, Cairo, Egypt

Ali Mahmoud Ahmed

Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Nguyen Tien Huy

Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.

Corresponding author

Correspondence to Nguyen Tien Huy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)

Additional file 2:

Figure S2. Risk of bias assessment summary. (TIF 69 kb)

Additional file 3:

Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)

Additional file 4:

Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)

Additional file 5:

Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)

Additional file 6:

Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)

Additional file 7:

Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health 47 , 46 (2019). https://doi.org/10.1186/s41182-019-0165-6

Download citation

Received : 30 January 2019

Accepted : 24 May 2019

Published : 01 August 2019

DOI : https://doi.org/10.1186/s41182-019-0165-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Tropical Medicine and Health

ISSN: 1349-4147

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Advertisement

How to conduct systematic literature reviews in management research: a guide in 6 steps and 14 decisions

• Review Paper
• Open access
• Published: 12 May 2023
• Volume 17 , pages 1899–1933, ( 2023 )

Cite this article

You have full access to this open access article

• Philipp C. Sauer   ORCID: orcid.org/0000-0002-1823-0723 1 &
• Stefan Seuring   ORCID: orcid.org/0000-0003-4204-9948 2  

30k Accesses

70 Citations

6 Altmetric

Explore all metrics

Systematic literature reviews (SLRs) have become a standard tool in many fields of management research but are often considerably less stringently presented than other pieces of research. The resulting lack of replicability of the research and conclusions has spurred a vital debate on the SLR process, but related guidance is scattered across a number of core references and is overly centered on the design and conduct of the SLR, while failing to guide researchers in crafting and presenting their findings in an impactful way. This paper offers an integrative review of the widely applied and most recent SLR guidelines in the management domain. The paper adopts a well-established six-step SLR process and refines it by sub-dividing the steps into 14 distinct decisions: (1) from the research question, via (2) characteristics of the primary studies, (3) to retrieving a sample of relevant literature, which is then (4) selected and (5) synthesized so that, finally (6), the results can be reported. Guided by these steps and decisions, prior SLR guidelines are critically reviewed, gaps are identified, and a synthesis is offered. This synthesis elaborates mainly on the gaps while pointing the reader toward the available guidelines. The paper thereby avoids reproducing existing guidance but critically enriches it. The 6 steps and 14 decisions provide methodological, theoretical, and practical guidelines along the SLR process, exemplifying them via best-practice examples and revealing their temporal sequence and main interrelations. The paper guides researchers in the process of designing, executing, and publishing a theory-based and impact-oriented SLR.

Similar content being viewed by others

The burgeoning role of literature review articles in management research: an introduction and outlook

On being ‘systematic’ in literature reviews

On being ‘systematic’ in literature reviews in IS

Explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

The application of systematic or structured literature reviews (SLRs) has developed into an established approach in the management domain (Kraus et al. 2020 ), with 90% of management-related SLRs published within the last 10 years (Clark et al. 2021 ). Such reviews help to condense knowledge in the field and point to future research directions, thereby enabling theory development (Fink 2010 ; Koufteros et al. 2018 ). SLRs have become an established method by now (e.g., Durach et al. 2017 ; Koufteros et al. 2018 ). However, many SLR authors struggle to efficiently synthesize and apply review protocols and justify their decisions throughout the review process (Paul et al. 2021 ) since only a few studies address and explain the respective research process and the decisions to be taken in this process. Moreover, the available guidelines do not form a coherent body of literature but focus on the different details of an SLR, while a comprehensive and detailed SLR process model is lacking. For example, Seuring and Gold ( 2012 ) provide some insights into the overall process, focusing on content analysis for data analysis without covering the practicalities of the research process in detail. Similarly, Durach et al. ( 2017 ) address SLRs from a paradigmatic perspective, offering a more foundational view covering ontological and epistemological positions. Durach et al. ( 2017 ) emphasize the philosophy of science foundations of an SLR. Although somewhat similar guidelines for SLRs might be found in the wider body of literature (Denyer and Tranfield 2009 ; Fink 2010 ; Snyder 2019 ), they often take a particular focus and are less geared toward explaining and reflecting on the single choices being made during the research process. The current body of SLR guidelines leaves it to the reader to find the right links among the guidelines and to justify their inconsistencies. This is critical since a vast number of SLRs are conducted by early-stage researchers who likely struggle to synthesize the existing guidance and best practices (Fisch and Block 2018 ; Kraus et al. 2020 ), leading to the frustration of authors, reviewers, editors, and readers alike.

Filling these gaps is critical in our eyes since researchers conducting literature reviews form the foundation of any kind of further analysis to position their research into the respective field (Fink 2010 ). So-called “systematic literature reviews” (e.g., Davis and Crombie 2001 ; Denyer and Tranfield 2009 ; Durach et al. 2017 ) or “structured literature reviews” (e.g., Koufteros et al. 2018 ; Miemczyk et al. 2012 ) differ from nonsystematic literature reviews in that the analysis of a certain body of literature becomes a means in itself (Kraus et al. 2020 ; Seuring et al. 2021 ). Although two different terms are used for this approach, the related studies refer to the same core methodological references that are also cited in this paper. Therefore, we see them as identical and abbreviate them as SLR.

There are several guidelines on such reviews already, which have been developed outside the management area (e.g. Fink 2010 ) or with a particular focus on one management domain (e.g., Kraus et al. 2020 ). SLRs aim at capturing the content of the field at a point in time but should also aim at informing future research (Denyer and Tranfield 2009 ), making follow-up research more efficient and productive (Kraus et al. 2021 ). Such standalone literature reviews would and should also prepare subsequent empirical or modeling research, but usually, they require far more effort and time (Fisch and Block 2018 ; Lim et al. 2022 ). To achieve this preparation, SLRs can essentially a) describe the state of the literature, b) test a hypothesis based on the available literature, c) extend the literature, and d) critique the literature (Xiao and Watson 2019 ). Beyond guiding the next incremental step in research, SLRs “may challenge established assumptions and norms of a given field or topic, recognize critical problems and factual errors, and stimulate future scientific conversations around that topic” (Kraus et al. 2022 , p. 2578). Moreover, they have the power to answer research questions that are beyond the scope of individual empirical or modeling studies (Snyder 2019 ) and to build, elaborate, and test theories beyond this single study scope (Seuring et al. 2021 ). These contributions of an SLR may be highly influential and therefore underline the need for high-quality planning, execution, and reporting of their process and details.

Regardless of the individual aims of standalone SLRs, their numbers have exponentially risen in the last two decades (Kraus et al. 2022 ) and almost all PhD or large research project proposals in the management domain include such a standalone SLR to build a solid foundation for their subsequent work packages. Standalone SLRs have thus become a key part of management research (Kraus et al. 2021 ; Seuring et al. 2021 ), which is also underlined by the fact that there are journals and special issues exclusively accepting standalone SLRs (Kraus et al. 2022 ; Lim et al. 2022 ).

However, SLRs require a commitment that is often comparable to an additional research process or project. Hence, SLRs should not be taken as a quick solution, as a simplistic, descriptive approach would usually not yield a publishable paper (see also Denyer and Tranfield 2009 ; Kraus et al. 2020 ).

Furthermore, as with other research techniques, SLRs are based on the rigorous application of rules and procedures, as well as on ensuring the validity and reliability of the method (Fisch and Block 2018 ; Seuring et al. 2021 ). In effect, there is a need to ensure “the same level of rigour to reviewing research evidence as should be used in producing that research evidence in the first place” (Davis and Crombie 2001 , p.1). This rigor holds for all steps of the research process, such as establishing the research question, collecting data, analyzing it, and making sense of the findings (Durach et al. 2017 ; Fink 2010 ; Seuring and Gold 2012 ). However, there is a high degree of diversity where some would be justified, while some papers do not report the full details of the research process. This lack of detail contrasts with an SLR’s aim of creating a valid map of the currently available research in the reviewed field, as critical information on the review’s completeness and potential reviewer biases cannot be judged by the reader or reviewer. This further impedes later replications or extensions of such reviews, which could provide longitudinal evidence of the development of a field (Denyer and Tranfield 2009 ; Durach et al. 2017 ). Against this observation, this paper addresses the following question:

Which decisions need to be made in an SLR process, and what practical guidelines can be put forward for making these decisions?

Answering this question, the key contributions of this paper are fourfold: (1) identifying the gaps in existing SLR guidelines, (2) refining the SLR process model by Durach et al. ( 2017 ) through 14 decisions, (3) synthesizing and enriching guidelines for these decisions, exemplifying the key decisions by means of best practice SLRs, and (4) presenting and discussing a refined SLR process model.

In some cases, we point to examples from operations and supply chain management. However, they illustrate the purposes discussed in the respective sections. We carefully checked that the arguments held for all fields of management-related research, and multiple examples from other fields of management were also included.

2 Identification of the need for an enriched process model, including a set of sequential decisions and their interrelations

In line with the exponential increase in SLR papers (Kraus et al. 2022 ), multiple SLR guidelines have recently been published. Since 2020, we have found a total of 10 papers offering guidelines on SLRs and other reviews for the field of management in general or some of its sub-fields. These guidelines are of double interest to this paper since we aim to complement them to fill the gap identified in the introduction while minimizing the doubling of efforts. Table 1 lists the 10 most recent guidelines and highlights their characteristics, research objectives, contributions, and how our paper aims to complement these previous contributions.

The sheer number and diversity of guideline papers, as well as the relevance expressed in them, underline the need for a comprehensive and exhaustive process model. At the same time, the guidelines take specific foci on, for example, updating earlier guidelines to new technological potentials (Kraus et al. 2020 ), clarifying the foundational elements of SLRs (Kraus et al. 2022 ) and proposing a review protocol (Paul et al. 2021 ) or the application and development of theory in SLRs (Seuring et al. 2021 ). Each of these foci fills an entire paper, while the authors acknowledge that much more needs to be considered in an SLR. Working through these most recent guidelines, it becomes obvious that the common paper formats in the management domain create a tension for guideline papers between elaborating on a) the SLR process and b) the details, options, and potentials of individual process steps.

Our analysis in Table 1 evidences that there are a number of rich contributions on aspect b), while the aspect a) of SLR process models has not received the same attention despite the substantial confusion of authors toward them (Paul et al. 2021 ). In fact, only two of the most recent guidelines approach SLR process models. First, Kraus et al. ( 2020 ) incrementally extended the 20-year-old Tranfield et al. ( 2003 ) three-stage model into four stages. A little later, Paul et al. ( 2021 ) proposed a three-stage (including six sub-stages) SPAR-4-SLR review protocol. It integrates the PRISMA reporting items (Moher et al. 2009 ; Page et al. 2021 ) that originate from clinical research to define 14 actions stating what items an SLR in management needs to report for reasons of validity, reliability, and replicability. Almost naturally, these 14 reporting-oriented actions mainly relate to the first SLR stage of “assembling the literature,” which accounts for nine of the 14 actions. Since this protocol is published in a special issue editorial, its presentation and elaboration are somewhat limited by the already mentioned word count limit. Nevertheless, the SPAR-4-SLR protocol provides a very useful checklist for researchers that enables them to include all data required to document the SLR and to avoid confusion from editors, reviewers, and readers regarding SLR characteristics.

Beyond Table 1 , Durach et al. ( 2017 ) synthesized six common SLR “steps” that differ only marginally in the delimitation of one step to another from the sub-stages of the previously mentioned SLR processes. In addition, Snyder ( 2019 ) proposed a process comprising four “phases” that take more of a bird’s perspective in addressing (1) design, (2) conduct, (3) analysis, and (4) structuring and writing the review. Moreover, Xiao and Watson ( 2019 ) proposed only three “stages” of (1) planning, (2) conducting, and (3) reporting the review that combines the previously mentioned conduct and the analysis and defines eight steps within them. Much in line with the other process models, the final reporting stage contains only one of the eight steps, leaving the reader somewhat alone in how to effectively craft a manuscript that contributes to the further development of the field.

In effect, the mentioned SLR processes differ only marginally, while the systematic nature of actions in the SPAR-4-SLR protocol (Paul et al. 2021 ) can be seen as a reporting must-have within any of the mentioned SLR processes. The similarity of the SLR processes is, however, also evident in the fact that they leave open how the SLR analysis can be executed, enriched, and reflected to make a contribution to the reviewed field. In contrast, this aspect is richly described in the other guidelines that do not offer an SLR process, leading us again toward the tension for guideline papers between elaborating on a) the SLR process and b) the details, options, and potentials of each process step.

To help (prospective) SLR authors successfully navigate this tension of existing guidelines, it is thus the ambition of this paper to adopt a comprehensive SLR process model along which an SLR project can be planned, executed, and written up in a coherent way. To enable this coherence, 14 distinct decisions are defined, reflected, and interlinked, which have to be taken across the different steps of the SLR process. At the same time, our process model aims to actively direct researchers to the best practices, tips, and guidance that previous guidelines have provided for individual decisions. We aim to achieve this by means of an integrative review of the relevant SLR guidelines, as outlined in the following section.

3 Methodology: an integrative literature review of guidelines for systematic literature reviews in management

It might seem intuitive to contribute to the debate on the “gold standard” of systematic literature reviews (Davis et al. 2014 ) by conducting a systematic review ourselves. However, there are different types of reviews aiming for distinctive contributions. Snyder ( 2019 ) distinguished between a) systematic, b) semi-systematic, and c) integrative (or critical) reviews, which aim for i) (mostly quantitative) synthesis and comparison of prior (primary) evidence, ii) an overview of the development of a field over time, and iii) a critique and synthesis of prior perspectives to reconceptualize or advance them. Each review team needs to position itself in such a typology of reviews to define the aims and scope of the review. To do so and structure the related research process, we adopted the four generic steps for an (integrative) literature review by Snyder ( 2019 )—(1) design, (2) conduct, (3) analysis, and (4) structuring and writing the review—on which we report in the remainder of this section. Since the last step is a very practical one that, for example, asks, “Is the contribution of the review clearly communicated?” (Snyder 2019 ), we will focus on the presentation of the method applied to the initial three steps:

(1) Regarding the design, we see the need for this study emerging from our experience in reviewing SLR manuscripts, supervising PhD students who, almost by default, need to prepare an SLR, and recurring discussions on certain decisions in the process of both. These discussions regularly left some blank or blurry spaces (see Table 1 ) that induced substantial uncertainty regarding critical decisions in the SLR process (Paul et al. 2021 ). To address this gap, we aim to synthesize prior guidance and critically enrich it, thus adopting an integrative approach for reviewing existing SLR guidance in the management domain (Snyder 2019 ).

(2) To conduct the review, we started collecting the literature that provided guidance on the individual SLR parts. We built on a sample of 13 regularly cited or very recent papers in the management domain. We started with core articles that we successfully used to publish SLRs in top-tier OSCM journals, such as Tranfield et al. ( 2003 ) and Durach et al. ( 2017 ), and we checked their references and papers that cited these publications. The search focus was defined by the following criteria: the articles needed to a) provide original methodological guidance for SLRs by providing new aspects of the guideline or synthesizing existing ones into more valid guidelines and b) focus on the management domain. Building on the nature of a critical or integrative review that does not require a full or representative sample (Snyder 2019 ), we limited the sample to the papers displayed in Table 2 that built the core of the currently applied SLR guidelines. In effect, we found 11 technical papers and two SLRs of SLRs (Carter and Washispack 2018 ; Seuring and Gold 2012 ). From the latter, we mainly analyzed the discussion and conclusion parts that explicitly developed guidance on conducting SLRs.

(3) For analyzing these papers, we first adopted the six-step SLR process proposed by Durach et al. ( 2017 , p.70), which they define as applicable to any “field, discipline or philosophical perspective”. The contrast between the six-step SLR process used for the analysis and the four-step process applied by ourselves may seem surprising but is justified by the use of an integrative approach. This approach differs mainly in retrieving and selecting pertinent literature that is key to SLRs and thus needs to be part of the analysis framework.

While deductively coding the sample papers against Durach et al.’s ( 2017 ), guidance in the six steps, we inductively built a set of 14 decisions presented in the right columns of Table 2 that are required to be made in any SLR. These decisions built a second and more detailed level of analysis, for which the single guidelines were coded as giving low, medium, or high levels of detail (see Table 3 ), which helped us identify the gaps in the current guidance papers and led our way in presenting, critically discussing, and enriching the literature. In effect, we see that almost all guidelines touch on the same issues and try to give a comprehensive overview. However, this results in multiple guidelines that all lack the space to go into detail, while only a few guidelines focus on filling a gap in the process. It is our ambition with this analysis to identify the gaps in the guidelines, thereby identifying a precise need for refinement, and to offer a first step into this refinement. Adopting advice from the literature sample, the coding was conducted by the entire author team (Snyder 2019 ; Tranfield et al. 2003 ) including discursive alignments of interpretation (Seuring and Gold 2012 ). This enabled a certain reliability and validity of the analysis by reducing the within-study and expectancy bias (Durach et al. 2017 ), while the replicability was supported by reporting the review sample and the coding results in Table 3 (Carter and Washispack 2018 ).

(4) For the writing of the review, we only pointed to the unusual structure of presenting the method without a theory section and then the findings in the following section. However, this was motivated by the nature of the integrative review so that the review findings at the same time represent the “state of the art,” “literature review,” or “conceptualization” sections of a paper.

4 Findings of the integrative review: presentation, critical discussion, and enrichment of prior guidance

4.1 the overall research process for a systematic literature review.

Even within our sample of only 13 guidelines, there are four distinct suggestions for structuring the SLR process. One of the earliest SLR process models was proposed by Tranfield et al. ( 2003 ) encompassing the three stages of (1) planning the review, (2) conducting a review, and (3) reporting and dissemination. Snyder ( 2019 ) proposed four steps employed in this study: (1) design, (2) conduct, (3) analysis, and (4) structuring and writing the review. Borrowing from content analysis guidelines, Seuring and Gold ( 2012 ) defined four steps: (1) material collection, (2) descriptive analysis, (3) category selection, and (4) material evaluation. Most recently Kraus et al. ( 2020 ) proposed four steps: (1) planning the review, (2) identifying and evaluating studies, (3) extracting and synthesizing data, and (4) disseminating the review findings. Most comprehensively, Durach et al. ( 2017 ) condensed prior process models into their generic six steps for an SLR. Adding the review of the process models by Snyder ( 2019 ) and Seuring and Gold ( 2012 ) to Durach et al.’s ( 2017 ) SLR process review of four papers, we support their conclusion of the general applicability of the six steps defined. Consequently, these six steps form the backbone of our coding scheme, as shown in the left column of Table 2 and described in the middle column.

As stated in Sect.  3 , we synthesized the review papers against these six steps but experienced that the papers were taking substantially different foci by providing rich details for some steps while largely bypassing others. To capture this heterogeneity and better operationalize the SLR process, we inductively introduced the right column, identifying 14 decisions to be made. These decisions are all elaborated in the reviewed papers but to substantially different extents, as the detailed coding results in Table 3 underline.

Mapping Table 3 for potential gaps in the existing guidelines, we found six decisions on which we found only low- to medium-level details, while high-detail elaboration was missing. These six decisions, which are illustrated in Fig.  1 , belong to three steps: 1: defining the research question, 5: synthesizing the literature, and 6: reporting the results. This result underscores our critique of currently unbalanced guidance that is, on the one hand, detailed on determining the required characteristics of primary studies (step 2), retrieving a sample of potentially relevant literature (step 3), and selecting the pertinent literature (step 4). On the other hand, authors, especially PhD students, are left without substantial guidance on the steps critical to publication. Instead, they are called “to go one step further … and derive meaningful conclusions” (Fisch and Block 2018 , p. 105) without further operationalizations on how this can be achieved; for example, how “meet the editor” conference sessions regularly cause frustration among PhDs when editors call for “new,” “bold,” and “relevant” research. Filling the gaps in the six decisions with best practice examples and practical experience is the main focus of this study’s contribution. The other eight decisions are synthesized with references to the guidelines that are most helpful and relevant for the respective step in our eyes.

The 6 steps and 14 decisions of the SLR process

4.2 Step 1: defining the research question

When initiating a research project, researchers make three key decisions.

Decision 1 considers the essential tasks of establishing a relevant and timely research question, but despite the importance of the decision, which determines large parts of further decisions (Snyder 2019 ; Tranfield et al. 2003 ), we only find scattered guidance in the literature. Hence, how can a research topic be specified to allow a strong literature review that is neither too narrow nor too broad? The latter is the danger in meta-reviews (i.e., reviews of reviews) (Aguinis et al. 2020 ; Carter and Washispack 2018 ; Kache and Seuring 2014 ). In this respect, even though the method would be robust, the findings would not be novel. In line with Carter and Washispack ( 2018 ), there should always be room for new reviews, yet over time, they must move from a descriptive overview of a field further into depth and provide detailed analyses of constructs. Clark et al. ( 2021 ) provided a detailed but very specific reflection on how they crafted a research question for an SLR and that revisiting the research question multiple times throughout the SLR process helps to coherently and efficiently move forward with the research. More generically, Kraus et al. ( 2020 ) listed six key contributions of an SLR that can guide the definition of the research question. Finally, Snyder ( 2019 ) suggested moving into more detail from existing SLRs and specified two main avenues for crafting an SLR research question that are either investigating the relationship among multiple effects, the effect of (a) specific variable(s), or mapping the evidence regarding a certain research area. For the latter, we see three possible alternative approaches, starting with a focus on certain industries. Examples are analyses of the food industry (Beske et al. 2014 ), retailing (Wiese et al. 2012 ), mining and minerals (Sauer and Seuring 2017 ), or perishable product supply chains (Lusiantoro et al. 2018 ) and traceability at the example of the apparel industry (Garcia-Torres et al. 2019 ). A second opportunity would be to assess the status of research in a geographical area that composes an interesting context from a research perspective, such as sustainable supply chain management (SSCM) in Latin America (Fritz and Silva 2018 ), yet this has to be justified explicitly, avoiding the fact that geographical focus is taken as the reason per se (e.g., Crane et al. 2016 ). A third variant addresses emerging issues, such as SCM, in a base-of-the-pyramid setting (Khalid and Seuring 2019 ) and the use of blockchain technology (Wang et al. 2019 ) or digital transformation (Hanelt et al. 2021 ). These approaches limit the reviewed field to enable a more contextualized analysis in which the novelty, continued relevance, or unjustified underrepresentation of the context can be used to specify a research gap and related research question(s). This also impacts the following decisions, as shown below.

Decision 2 concerns the option for a theoretical approach (i.e., the adoption of an inductive, abductive, or deductive approach) to theory building through the literature review. The review of previous guidance on this delivers an interesting observation. On the one hand, there are early elaborations on systematic reviews, realist synthesis, meta-synthesis, and meta-analysis by Tranfield et al. ( 2003 ) that are borrowing from the origins of systematic reviews in medical research. On the other hand, recent management-related guidelines largely neglect details of related decisions, but point out that SLRs are a suitable tool for theory building (Kraus et al. 2020 ). Seuring et al. ( 2021 ) set out to fill this gap and provided substantial guidance on how to use theory in SLRs to advance the field. To date, the option for a theoretical approach is only rarely made explicit, leaving the reader often puzzled about how advancement in theory has been crafted and impeding a review’s replicability (Seuring et al. 2021 ). Many papers still leave related choices in the dark (e.g., Rhaiem and Amara 2021 ; Rojas-Córdova et al. 2022 ) and move directly from the introduction to the method section.

In Decision 3, researchers need to adopt a theoretical framework (Durach et al. 2017 ) or at least a theoretical starting point, depending on the most appropriate theoretical approach (Seuring et al. 2021 ). Here, we find substantial guidance by Durach et al. ( 2017 ) that underlines the value of adopting a theoretical lens to investigate SCM phenomena and the literature. Moreover, the choice of a theoretical anchor enables a consistent definition and operationalization of constructs that are used to analyze the reviewed literature (Durach et al. 2017 ; Seuring et al. 2021 ). Hence, providing some upfront definitions is beneficial, clarifying what key terminology would be used in the subsequent paper, such as Devece et al. ( 2019 ) introduce their terminology on coopetition. Adding a practical hint beyond the elaborations of prior guidance papers for taking up established constructs in a deductive analysis (decision 2), there would be the question of whether these can yield interesting findings.

Here, it would be relevant to specify what kind of analysis is aimed for the SLR, where three approaches might be distinguished (i.e., bibliometric analysis, meta-analysis, and content analysis–based studies). Briefly distinguishing them, the core difference would be how many papers can be analyzed employing the respective method. Bibliometric analysis (Donthu et al. 2021 ) usually relies on the use of software, such as Biblioshiny, allowing the creation of figures on citations and co-citations. These figures enable the interpretation of large datasets in which several hundred papers can be analyzed in an automated manner. This allows for distinguishing among different research clusters, thereby following a more inductive approach. This would be contrasted by meta-analysis (e.g., Leuschner et al. 2013 ), where often a comparatively smaller number of papers is analyzed (86 in the respective case) but with a high number of observations (more than 17,000). The aim is to test for statistically significant correlations among single constructs, which requires that the related constructs and items be precisely defined (i.e., a clearly deductive approach to the analysis).

Content analysis is the third instrument frequently applied to data analysis, where an inductive or deductive approach might be taken (Seuring et al. 2021 ). Content-based analysis (see decision 9 in Sect.  4.6 ; Seuring and Gold 2012 ) is a labor-intensive step and can hardly be changed ex post. This also implies that only a certain number of papers might be analyzed (see Decision 6 in Sect.  4.5 ). It is advisable to adopt a wider set of constructs for the analysis stemming even from multiple established frameworks since it is difficult to predict which constructs and items might yield interesting insights. Hence, coding a more comprehensive set of items and dropping some in the process is less problematic than starting an analysis all over again for additional constructs and items. However, in the process of content analysis, such an iterative process might be required to improve the meaningfulness of the data and findings (Seuring and Gold 2012 ). A recent example of such an approach can be found in Khalid and Seuring ( 2019 ), building on the conceptual frameworks for SSCM of Carter and Rogers ( 2008 ), Seuring and Müller ( 2008 ), and Pagell and Wu ( 2009 ). This allows for an in-depth analysis of how SSCM constructs are inherently referred to in base-of-the-pyramid-related research. The core criticism and limitation of such an approach is the random and subjectively biased selection of frameworks for the purpose of analysis.

Beyond the aforementioned SLR methods, some reviews, similar to the one used here, apply a critical review approach. This is, however, nonsystematic, and not an SLR; thus, it is beyond the scope of this paper. Interested readers can nevertheless find some guidance on critical reviews in the available literature (e.g., Kraus et al. 2022 ; Snyder 2019 ).

4.3 Step 2: determining the required characteristics of primary studies

After setting the stage for the review, it is essential to determine which literature is to be reviewed in Decision 4. This topic is discussed by almost all existing guidelines and will thus only briefly be discussed here. Durach et al. ( 2017 ) elaborated in great detail on defining strict inclusion and exclusion criteria that need to be aligned with the chosen theoretical framework. The relevant units of analysis need to be specified (often a single paper, but other approaches might be possible) along with suitable research methods, particularly if exclusively empirical studies are reviewed or if other methods are applied. Beyond that, they elaborated on potential quality criteria that should be applied. The same is considered by a number of guidelines that especially draw on medical research, in which systematic reviews aim to pool prior studies to infer findings from their total population. Here, it is essential to ensure the exclusion of poor-quality evidence that would lower the quality of the review findings (Mulrow 1987 ; Tranfield et al. 2003 ). This could be ensured by, for example, only taking papers from journals listed on the Web of Science or Scopus or journals listed in quartile 1 of Scimago ( https://www.scimagojr.com/ ), a database providing citation and reference data for journals.

The selection of relevant publication years should again follow the purpose of the study defined in Step 1. As such, there might be a justified interest in the wide coverage of publication years if a historical perspective is taken. Alternatively, more contemporary developments or the analysis of very recent issues can justify the selection of very few years of publication (e.g., Kraus et al. 2022 ). Again, it is hard to specify a certain time period covered, but if developments of a field should be analyzed, a five-year period might be a typical lower threshold. On current topics, there is often a trend of rising publishing numbers. This scenario implies the rising relevance of a topic; however, this should be treated with caution. The total number of papers published per annum has increased substantially in recent years, which might account for the recently heightened number of papers on a certain topic.

4.4 Step 3: retrieving a sample of potentially relevant literature

After defining the required characteristics of the literature to be reviewed, the literature needs to be retrieved based on two decisions. Decision 5 concerns suitable literature sources and databases that need to be defined. Turning to Web of Science or Scopus would be two typical options found in many of the examples mentioned already (see also detailed guidance by Paul and Criado ( 2020 ) as well as Paul et al. ( 2021 )). These databases aggregate many management journals, and a typical argument for turning to the Web of Science database is the inclusion of impact factors, as they indicate a certain minimum quality of the journal (Sauer and Seuring 2017 ). Additionally, Google Scholar is increasingly mentioned as a usable search engine, often providing higher numbers of search results than the mentioned databases (e.g., Pearce 2018 ). These results often entail duplicates of articles from multiple sources or versions of the same article, as well as articles in predatory journals (Paul et al. 2021 ). Therefore, we concur with Paul et al. ( 2021 ) who underline the quality assurance mechanisms in Web of Science and Scopus, making them preferred databases for the literature search. From a practical perspective, it needs to be mentioned that SLRs in management mainly rely on databases that are not free to use. Against this limitation, Pearce ( 2018 ) provided a list of 20 search engines that are free of charge and elaborated on their advantages and disadvantages. Due to the individual limitations of the databases, it is advisable to use a combination of them (Kraus et al. 2020 , 2022 ) and build a consolidated sample by screening the papers found for duplicates, as regularly done in SLRs.

This decision also includes the choice of the types of literature to be analyzed. Typically, journal papers are selected, ensuring that the collected papers are peer-reviewed and have thus undergone an academic quality management process. Meanwhile, conference papers are usually avoided since they are often less mature and not checked for quality (e.g., Seuring et al. 2021 ). Nevertheless, for emerging topics, it might be too restrictive to consider only peer-reviewed journal articles and limit the literature to only a few references. Analyzing such rapidly emerging topics is relevant for timely and impact-oriented research and might justify the selection of different sources. Kraus et al. ( 2020 ) provided a discussion on the use of gray literature (i.e., nonacademic sources), and Sauer ( 2021 ) provided an example of a review of sustainability standards from a management perspective to derive implications for their application by managers on the one hand and for enhancing their applicability on the other hand.

Another popular way to limit the review sample is the restriction to a certain list of journals (Kraus et al. 2020 ; Snyder 2019 ). While this is sometimes favored by highly ranked journals, Carter and Washispack ( 2018 ), for example, found that many pertinent papers are not necessarily published in journals within the field. Webster and Watson ( 2002 ) quite tellingly cited a reviewer labeling the selection of top journals as an unjustified excuse for investigating the full body of relevant literature. Both aforementioned guidelines thus discourage the restriction to particular journals, a guidance that we fully support.

However, there is an argument to be made supporting the exclusion of certain lower-ranked journals. This can be done, for example, by using Scimago Journal quartiles ( www.scimagojr.com , last accessed 13. of April 2023) and restricting it to journals in the first quartile (e.g., Yavaprabhas et al. 2022 ). Other papers (e.g., Kraus et al. 2021 ; Rojas-Córdova et al. 2022 ) use certain journal quality lists to limit their sample. However, we argue for a careful check by the authors against the topic reviewed regarding what would be included and excluded.

Decision 6 entails the definition of search terms and a search string to be applied in the database just chosen. The search terms should reflect the aims of the review and the exclusion criteria that might be derived from the unit of analysis and the theoretical framework (Durach et al. 2017 ; Snyder 2019 ). Overall, two approaches to keywords can be observed. First, some guides suggest using synonyms of the key terms of interest (e.g., Durach et al. 2017 ; Kraus et al. 2020 ) in order to build a wide baseline sample that will be condensed in the next step. This is, of course, especially helpful if multiple terms delimitate a field together or different synonymous terms are used in parallel in different fields or journals. Empirical journals in supply chain management, for example, use the term “multiple supplier tiers ” (e.g., Tachizawa and Wong 2014 ), while modeling journals in the same field label this as “multiple supplier echelons ” (e.g., Brandenburg and Rebs 2015 ). Second, in some cases, single keywords are appropriate for capturing a central aspect or construct of a field if the single keyword has a global meaning tying this field together. This approach is especially relevant to the study of relatively broad terms, such as “social media” (Lim and Rasul 2022 ). However, this might result in very high numbers of publications found and therefore requires a purposeful combination with other search criteria, such as specific journals (Kraus et al. 2021 ; Lim et al. 2021 ), publication dates, article types, research methods, or the combination with keywords covering domains to which the search is aimed to be specified.

Since SLRs are often required to move into detail or review the intersections of relevant fields, we recommend building groups of keywords (single terms or multiple synonyms) for each field to be connected that are coupled via Boolean operators. To determine when a point of saturation for a keyword group is reached, one could monitor the increase in papers found in a database when adding another synonym. Once the increase is significantly decreasing or even zeroing, saturation is reached (Sauer and Seuring 2017 ). The keywords themselves can be derived from the list of keywords of influential publications in the field, while attention should be paid to potential synonyms in neighboring fields (Carter and Washispack 2018 ; Durach et al. 2017 ; Kraus et al. 2020 ).

4.5 Step 4: selecting the pertinent literature

The inclusion and exclusion criteria (Decision 6) are typically applied in Decision 7 in a two-stage process, first on the title, abstract, and keywords of an article before secondly applying them to the full text of the remaining articles (see also Kraus et al. 2020 ; Snyder 2019 ). Beyond this, Durach et al. ( 2017 ) underlined that the pertinence of the publication regarding units of analysis and the theoretical framework needs to be critically evaluated in this step to avoid bias in the review analysis. Moreover, Carter and Washispack ( 2018 ) requested the publication of the included and excluded sources to ensure the replicability of Steps 3 and 4. This can easily be done as an online supplement to an eventually published review article.

Nevertheless, the question remains: How many papers justify a literature review? While it is hard to specify how many papers comprise a body of literature, there might be certain thresholds for which Kraus et al. ( 2020 ) provide a useful discussion. As a rough guide, more than 50 papers would usually make a sound starting point (see also Paul and Criado 2020 ), while there are SLRs on emergent topics, such as multitier supply chain management, where 39 studies were included (Tachizawa and Wong 2014 ). An SLR on “learning from innovation failures” builds on 36 papers (Rhaiem and Amara 2021 ), which we would see as the lower threshold. However, such a low number should be an exception, and anything lower would certainly trigger the following question: Why is a review needed? Meanwhile, there are also limits on how many papers should be reviewed. While there are cases with 191 (Seuring and Müller 2008 ), 235 (Rojas-Córdova et al. 2022 ), or up to nearly 400 papers reviewed (Spens and Kovács 2006 ), these can be regarded as upper thresholds. Over time, similar topics seem to address larger datasets.

4.6 Step 5: synthesizing the literature

Before synthesizing the literature, Decision 8 considers the selection of a data extraction tool for which we found surprisingly little guidance. Some guidance is given on the use of cloud storage to enable remote team work (Clark et al. 2021 ). Beyond this, we found that SLRs have often been compiled with marked and commented PDFs or printed papers that were accompanied by tables (Kraus et al. 2020 ) or Excel sheets (see also the process tips by Clark et al. 2021 ). This sheet tabulated the single codes derived from the theoretical framework (Decision 3) and the single papers to be reviewed (Decision 7) by crossing out individual cells, signaling the representation of a particular code in a particular paper. While the frequency distribution of the codes is easily compiled from this data tool, the related content needs to be looked at in the papers in a tedious back-and-forth process. Beyond that, we would strongly recommend using data analysis software, such as MAXQDA or NVivo. Such programs enable the import of literature in PDF format and the automatic or manual coding of text passages, their comparison, and tabulation. Moreover, there is a permanent and editable reference of the coded text to a code. This enables a very quick compilation of content summaries or statistics for single codes and the identification of qualitative and quantitative links between codes and papers.

All the mentioned data extraction or data processing tools require a license and therefore are not free of cost. While many researchers may benefit from national or institutional subscriptions to these services, others may not. As a potential alternative, Pearce ( 2018 ) proposed a set of free open-source software (FOSS), including an elaboration on how they can be combined to perform an SLR. He also highlighted that both free and proprietary solutions have advantages and disadvantages that are worthwhile for those who do not have the required tools provided by their employers or other institutions they are members of. The same may apply to the literature databases used for the literature acquisition in Decision 5 (Pearce 2018 ).

Moreover, there is a link to Step 1, Decision 3, where bibliometric reviews and meta-analyses were mentioned. These methods, which are alternatives to content analysis–based approaches, have specific demands, so specific tools would be appropriate, such as the Biblioshiny software or VOSviewer. As we will point out for all decisions, there is a high degree of interdependence among the steps and decisions made.

Decision 9 looks at conducting the data analysis, such as coding against (pre-defined) constructs, in SLRs that rely, in most cases, on content analysis. Seuring and Gold ( 2012 ) elaborated in detail on its characteristics and application in SLRs. As this paper also explains the process of qualitative content analysis in detail, repetition is avoided here, but a summary is offered. Since different ways exist to conduct a content analysis, it is even more important to explain and justify, for example, the choice of an inductive or deductive approach (see Decision 2). In several cases, analytic variables are applied on the go, so there is no theory-based introduction of related constructs. However, to ensure the validity and replicability of the review (see Decision 11), it is necessary to explicitly define all the variables and codes used to analyze and synthesize the reviewed material (Durach et al. 2017 ; Seuring and Gold 2012 ). To build a valid framework as the SLR outcome, it is vital to ensure that the constructs used for the data analysis are sufficiently defined, mutually exclusive, and comprehensively exhaustive. For meta-analysis, the predefined constructs and items would demand quantitative coding so that the resulting data could be analyzed using statistical software tools such as SPSS or R (e.g., Xiao and Watson 2019 ). Pointing to bibliometric analysis again, the respective software would be used for data analysis, yielding different figures and paper clusters, which would then require interpretation (e.g., Donthu et al. 2021 ; Xiao and Watson 2019 ).

Decision 10, on conducting subsequent statistical analysis, considers follow-up analysis of the coding results. Again, this is linked to the chosen SLR method, and a bibliographic analysis will require a different statistical analysis than a content analysis–based SLR (e.g., Lim et al. 2022 ; Xiao and Watson 2019 ). Beyond the use of content analysis and the qualitative interpretation of its results, applying contingency analysis offers the opportunity to quantitatively assess the links among constructs and items. It provides insights into which items are correlated with each other without implying causality. Thus, the interpretation of the findings must explain the causality behind the correlations between the constructs and the items. This must be based on sound reasoning and linking the findings to theoretical arguments. For SLRs, there have recently been two kinds of applications of contingency analysis, differentiated by unit of analysis. De Lima et al. ( 2021 ) used the entire paper as the unit of analysis, deriving correlations on two constructs that were used together in one paper. This is, of course, subject to critique as to whether the constructs really represent correlated content. Moving a level deeper, Tröster and Hiete ( 2018 ) used single-text passages on one aspect, argument, or thought as the unit of analysis. Such an approach is immune against the critique raised before and can yield more valid statistical support for thematic analysis. Another recent methodological contribution employing the same contingency analysis–based approach was made by Siems et al. ( 2021 ). Their analysis employs constructs from SSCM and dynamic capabilities. Employing four subsets of data (i.e., two time periods each in the food and automotive industries), they showed that the method allows distinguishing among time frames as well as among industries.

However, the unit of analysis must be precisely explained so that the reader can comprehend it. Both examples use contingency analysis to identify under-researched topics and develop them into research directions whose formulation represents the particular aim of an SLR (Paul and Criado 2020 ; Snyder 2019 ). Other statistical tools might also be applied, such as cluster analysis. Interestingly, Brandenburg and Rebs ( 2015 ) applied both contingency and cluster analyses. However, the authors stated that the contingency analysis did not yield usable results, so they opted for cluster analysis. In effect, Brandenburg and Rebs ( 2015 ) added analytical depth to their analysis of model types in SSCM by clustering them against the main analytical categories of content analysis. In any case, the application of statistical tools needs to fit the study purpose (Decision 1) and the literature sample (Decision 7), just as in their more conventional applications (e.g., in empirical research processes).

Decision 11 regards the additional consideration of validity and reliability criteria and emphasizes the need for explaining and justifying the single steps of the research process (Seuring and Gold 2012 ), much in line with other examples of research (Davis and Crombie 2001 ). This is critical to underlining the quality of the review but is often neglected in many submitted manuscripts. In our review, we find rich guidance on this decision, to which we want to guide readers (see Table 3 ). In particular, Durach et al. ( 2017 ) provide an entire section of biases and what needs to be considered and reported on them. Moreover, Snyder ( 2019 ) regularly reflects on these issues in her elaborations. This rich guidance elaborates on how to ensure the quality of the individual steps of the review process, such as sampling, study inclusion and exclusion, coding, synthesizing, and more practical issues, including team composition and teamwork organization, which are discussed in some guidelines (e.g., Clark et al. 2021 ; Kraus et al. 2020 ). We only want to underline that the potential biases are, of course, to be seen in conjunction with Decisions 2, 3, 4, 5, 6, 7, 9, and 10. These decisions and the elaboration by Durach et al. ( 2017 ) should provide ample points of reflection that, however, many SLR manuscripts fail to address.

4.7 Step 6: reporting the results

In the final step, there are three decisions on which there is surprisingly little guidance, although reviews often fail in this critical part of the process (Kraus et al. 2020 ). The reviewed guidelines discuss the presentation almost exclusively, while almost no guidance is given on the overall paper structure or the key content to be reported.

Consequently, the first choice to be made in Decision 12 is regarding the paper structure. We suggest following the five-step logic of typical research papers (see also Fisch and Block 2018 ) and explaining only a few points in which a difference from other papers is seen.

(1) Introduction: While the introduction would follow a conventional logic of problem statement, research question, contribution, and outline of the paper (see also Webster and Watson 2002 ), the next parts might depend on the theoretical choices made in Decision 2.

(2) Literature review section: If deductive logic is taken, the paper usually has a conventional flow. After the introduction, the literature review section covers the theoretical background and the choice of constructs and variables for the analysis (De Lima et al. 2021 ; Dieste et al. 2022 ). To avoid confusion in this section with the literature review, its labeling can also be closer to the reviewed object.

If an inductive approach is applied, it might be challenging to present the theoretical basis up front, as the codes emerge only from analyzing the material. In this case, the theory section might be rather short, concentrating on defining the core concepts or terms used, for example, in the keyword-based search for papers. The latter approach is exemplified by the study at hand, which presents a short review of the available literature in the introduction and the first part of the findings. However, we do not perform a systematic but integrative review, which allows for more freedom and creativity (Snyder 2019 ).

(3) Method section: This section should cover the steps and follow the logic presented in this paper or any of the reviewed guidelines so that the choices made during the research process are transparently disclosed (Denyer and Tranfield 2009 ; Paul et al. 2021 ; Xiao and Watson 2019 ). In particular, the search for papers and their selection requires a sound explanation of each step taken, including the provision of reasons for the delimitation of the final paper sample. A stage that is often not covered in sufficient detail is data analysis (Seuring and Gold 2012 ). This also needs to be outlined so that the reader can comprehend how sense has been made of the material collected. Overall, the demands for SLR papers are similar to case studies, survey papers, or almost any piece of empirical research; thus, each step of the research process needs to be comprehensively described, including Decisions 4–10. This comprehensiveness must also include addressing measures for validity and reliability (see Decision 11) or other suitable measures of rigor in the research process since they are a critical issue in literature reviews (Durach et al. 2017 ). In particular, inductively conducted reviews are prone to subjective influences and thus require sound reporting of design choices and their justification.

(4) Findings: The findings typically start with a descriptive analysis of the literature covered, such as journals, distribution across years, or (empirical) methods applied (Tranfield et al. 2003 ). For modeling-related reviews, classifying papers against the approach chosen is a standard approach, but this can often also serve as an analytic category that provides detailed insights. The descriptive analysis should be kept short since a paper only presenting descriptive findings will not be of great interest to other researchers due to the missing contribution (Snyder 2019 ). Nevertheless, there are opportunities to provide interesting findings in the descriptive analysis. Beyond a mere description of the distributions of the single results, such as the distribution of methods used in the sample, authors should combine analytical categories to derive more detailed insights (see also Tranfield et al. 2003 ). The distribution of methods used might well be combined with the years of publication to identify and characterize different phases in the development of a field of research or its maturity. Moreover, there could be value in the analysis of theories applied in the review sample (e.g., Touboulic and Walker 2015 ; Zhu et al. 2022 ) and in reflecting on the interplay of different qualitative and quantitative methods in spurring the theoretical development of the reviewed field. This could yield detailed insights into methodological as well as theoretical gaps, and we would suggest explicitly linking the findings of such analyses to the research directions that an SLR typically provides. This link could help make the research directions much more tangible by giving researchers a clear indication of how to follow up on the findings, as, for example, done by Maestrini et al. ( 2017 ) or Dieste et al. ( 2022 ). In contrast to the mentioned examples of an actionable research agenda, a typical weakness of premature SLR manuscripts is that they ask rather superficially for more research in the different aspects they reviewed but remain silent about how exactly this can be achieved.

We would thus like to encourage future SLR authors to systematically investigate the potential to combine two categories of descriptive analysis to move this section of the findings to a higher level of quality, interest, and relevance. The same can, of course, be done with the thematic findings, which comprise the second part of this section.

Moving into the thematic analysis, we have already reached Decision 13 on the presentation of the refined theoretical framework and the discussion of its contents. A first step might present the frequencies of the codes or constructs applied in the analysis. This allows the reader to understand which topics are relevant. If a rather small body of literature is analyzed, tables providing evidence on which paper has been coded for which construct might be helpful in improving the transparency of the research process. Tables or other forms of visualization might help to organize the many codes soundly (see also Durach et al. 2017 ; Paul and Criado 2020 ; Webster and Watson 2002 ). These findings might then lead to interpretation, for which it is necessary to extract meaning from the body of literature and present it accordingly (Snyder 2019 ). To do so, it might seem needless to say that the researchers should refer back to Decisions 1, 2, and 3 taken in Step 1 and their justifications. These typically identify the research gap to be filled, but after the lengthy process of the SLR, the authors often fail to step back from the coding results and put them into a larger perspective against the research gap defined in Decision 1 (see also Clark et al. 2021 ). To support this, it is certainly helpful to illustrate the findings in a figure or graph presenting the links among the constructs and items and adding causal reasoning to this (Durach et al. 2017 ; Paul and Criado 2020 ), such as the three figures by Seuring and Müller ( 2008 ) or other examples by De Lima et al. ( 2021 ) or Tipu ( 2022 ). This presentation should condense arguments made in the assessed literature but should also chart the course for future research. It will be these parts of the paper that are decisive for a strong SLR paper.

Moreover, some guidelines define the most fruitful way of synthesizing the findings as concept-centric synthesis (Clark et al. 2021 ; Fisch and Block 2018 ; Webster and Watson 2002 ). As presented in the previous sentence, the presentation of the review findings is centered on the content or concept of “concept-centric synthesis.” It is accompanied by a reference to all or the most relevant literature in which the concept is evident. Contrastingly, Webster and Watson ( 2002 ) found that author-centric synthesis discusses individual papers and what they have done and found (just like this sentence here). They added that this approach fails to synthesize larger samples. We want to note that we used the latter approach in some places in this paper. However, this aims to actively refer the reader to these studies, as they stand out from our relatively small sample. Beyond this, we want to link back to Decision 3, the selection of a theoretical framework and constructs. These constructs, or the parts of a framework, can also serve to structure the findings section by using them as headlines for subsections (Seuring et al. 2021 ).

Last but not least, there might even be cases where core findings and relationships might be opposed, and alternative perspectives could be presented. This would certainly be challenging to argue for but worthwhile to do in order to drive the reviewed field forward. A related example is the paper by Zhu et al. ( 2022 ), who challenged the current debate at the intersection of blockchain applications and supply chain management and pointed to the limited use of theoretical foundations for related analysis.

(5) Discussion and Conclusion: The discussion needs to explain the contribution the paper makes to the extant literature, that is, which previous findings or hypotheses are supported or contradicted and which aspects of the findings are particularly interesting for the future development of the reviewed field. This is in line with the content required in the discussion sections of any other paper type. A typical structure might point to the contribution and put it into perspective with already existing research. Further, limitations should be addressed on both the theoretical and methodological sides. This elaboration of the limitations can be coupled with the considerations of the validity and reliability of the study in Decision 11. The implications for future research are a core aim of an SLR (Clark et al. 2021 ; Mulrow 1987 ; Snyder 2019 ) and should be addressed in a further part of the discussion section. Recently, a growing number of literature reviews have also provided research questions for future research that provide a very concrete and actionable output of the SLR (e.g. Dieste et al. 2022 ; Maestrini et al. 2017 ). Moreover, we would like to reiterate our call to clearly link the research implications to the SLR findings, which helps the authors craft more tangible research directions and helps the reader to follow the authors’ interpretation. Literature review papers are usually not strongly positioned toward managerial implications, but even these implications might be included.

As a kind of normal demand, the conclusion should provide an answer to the research question put forward in the introduction, thereby closing the cycle of arguments made in the paper.

Although all the works seem to be done when the paper is written and the contribution is fleshed out, there is still one major decision to be made. Decision 14 concerns the identification of an appropriate journal for submission. Despite the popularity of the SLR method, a rising number of journals explicitly limit the number of SLRs published by them. Moreover, there are only two guidelines elaborating on this decision, underlining the need for the following considerations.

Although it might seem most attractive to submit the paper to the highest-ranking journal for the reviewed topic, we argue for two critical and review-related decisions to be made during the research process that influence whether the paper fits a certain outlet:

The theoretical foundation of the SLR (Decision 3) usually relates to certain journals in which it is published or discussed. If a deductive approach was taken, the journals in which the foundational papers were published might be suitable since the review potentially contributes to the further validation or refinement of the frameworks. Overall, we need to keep in mind that a paper needs to be added to a discussion in the journal, and this can be based on the theoretical framework or the reviewed papers, as shown below.

Appropriate journals for publication can be derived from the analyzed journal papers (Decision 7) (see also Paul and Criado 2020 ). This allows for an easy link to the theoretical debate in the respective journal by submitting it. This choice is identifiable in most of the papers mentioned in this paper and is often illustrated in the descriptive analysis.

If the journal chosen for the submission was neither related to the theoretical foundation nor overly represented in the body of literature analyzed, an explicit justification in the paper itself might be needed. Alternatively, an explanation might be provided in the letter to the editor when submitting the paper. If such a statement is not presented, the likelihood of it being transferred into the review process and passing it is rather low. Finally, we want to refer readers interested in the specificities of the publication-related review process of SLRs to Webster and Watson ( 2002 ), who elaborated on this for Management Information Systems Quarterly.

5 Discussion and conclusion

Critically reviewing the currently available SLR guidelines in the management domain, this paper synthesizes 14 key decisions to be made and reported across the SLR research process. Guidelines are presented for each decision, including tasks that assist in making sound choices to complete the research process and make meaningful contributions. Applying these guidelines should improve the rigor and robustness of many review papers and thus enhance their contributions. Moreover, some practical hints and best-practice examples are provided on issues that unexperienced authors regularly struggle to present in a manuscript (Fisch and Block 2018 ) and thus frustrate reviewers, readers, editors, and authors alike.

Strikingly, the review of prior guidelines reported in Table 3 revealed their focus on the technical details that need to be reported in any SLR. Consequently, our discipline has come a long way in crafting search strings, inclusion, and exclusion criteria, and elaborating on the validity and reliability of an SLR. Nevertheless, we left critical areas underdeveloped, such as the identification of relevant research gaps and questions, data extraction tools, analysis of the findings, and a meaningful and interesting reporting of the results. Our study contributes to filling these gaps by providing operationalized guidance to SLR authors, especially early-stage researchers who craft SLRs at the outset of their research journeys. At the same time, we need to underline that our paper is, of course, not the only useful reference for SLR authors. Instead, the readers are invited to find more guidance on the many aspects to consider in an SLR in the references we provide within the single decisions, as well as in Tables 1 and 2 . The tables also identify the strongholds of other guidelines that our paper does not want to replace but connect and extend at selected occasions, especially in SLR Steps 5 and 6.

The findings regularly underline the interconnection of the 14 decisions identified and discussed in this paper. We thus support Tranfield et al. ( 2003 ) who requested a flexible approach to the SLR while clearly reporting all design decisions and reflecting their impacts. In line with the guidance synthesized in this review, and especially Durach et al. ( 2017 ), we also present a refined framework in Figs.  1 and 2 . It specifically refines the original six-step SLR process by Durach et al. ( 2017 ) in three ways:

Enriched six-step process including the core interrelations of the 14 decisions

First, we subdivided the six steps into 14 decisions to enhance the operationalization of the process and enable closer guidance (see Fig.  1 ). Second, we added a temporal sequence to Fig.  2 by positioning the decisions from left to right according to this temporal sequence. This is based on systematically reflecting on the need to finish one decision before the following. If this need is evident, the following decision moves to the right; if not, the decisions are positioned below each other. Turning to Fig.  2 , it becomes evident that Step 2, “determining the required characteristics of primary studies,” and Step 3, “retrieving a sample of potentially relevant literature,” including their Decisions 4–6, can be conducted in an iterative manner. While this contrasts with the strict division of the six steps by Durach et al. ( 2017 ), it supports other guidance that suggests running pilot studies to iteratively define the literature sample, its sources, and characteristics (Snyder 2019 ; Tranfield et al. 2003 ; Xiao and Watson 2019 ). While this insight might suggest merging Steps 2 and 3, we refrain from this superficial change and building yet another SLR process model. Instead, we prefer to add detail and depth to Durach et al.’s ( 2017 ) model.

(Decisions: D1: specifying the research gap and related research question, D2: opting for a theoretical approach, D3: defining the core theoretical framework and constructs, D4: specifying inclusion and exclusion criteria, D5: defining sources and databases, D6: defining search terms and crafting a search string, D7: including and excluding literature for detailed analysis and synthesis, D8: selecting data extraction tool(s), D9: coding against (pre-defined) constructs, D10: conducting a subsequent (statistical) analysis (optional), D11: ensuring validity and reliability, D12: deciding on the structure of the paper, D13: presenting a refined theoretical framework and discussing its contents, and D14: deriving an appropriate journal from the analyzed papers).

This is also done through the third refinement, which underlines which previous or later decisions need to be considered within each single decision. Such a consideration moves beyond the mere temporal sequence of steps and decisions that does not reflect the full complexity of the SLR process. Instead, its focus is on the need to align, for example, the conduct of the data analysis (Decision 9) with the theoretical approach (Decision 2) and consequently ensure that the chosen theoretical framework and the constructs (Decision 3) are sufficiently defined for the data analysis (i.e., mutually exclusive and comprehensively exhaustive). The mentioned interrelations are displayed in Fig.  2 by means of directed arrows from one decision to another. The underlying explanations can be found in the earlier paper sections by searching for the individual decisions in the text on the impacted decisions. Overall, it is unsurprising to see that the vast majority of interrelations are directed from the earlier to the later steps and decisions (displayed through arrows below the diagonal of decisions), while only a few interrelations are inverse.

Combining the first refinement of the original framework (defining the 14 decisions) and the third refinement (revealing the main interrelations among the decisions) underlines the contribution of this study in two main ways. First, the centrality of ensuring validity and reliability (Decision 11) is underlined. It becomes evident that considerations of validity and reliability are central to the overall SLR process since all steps before the writing of the paper need to be revisited in iterative cycles through Decision 11. Any lack of related considerations will most likely lead to reviewer critique, putting the SLR publication at risk. On the positive side of this centrality, we also found substantial guidance on this issue. In contrast, as evidenced in Table 3 , there is a lack of prior guidance on Decisions 1, 8, 10, 12, 13, and 14, which this study is helping to fill. At the same time, these underexplained decisions are influenced by 14 of the 44 (32%) incoming arrows in Fig.  2 and influence the other decisions in 6 of the 44 (14%) instances. These interrelations among decisions to be considered when crafting an SLR were scattered across prior guidelines, lacked in-depth elaborations, and were hardly explicitly related to each other. Thus, we hope that our study and the refined SLR process model will help enhance the quality and contribution of future SLRs.

Data availablity

The data generated during this research is summarized in Table 3 and the analyzed papers are publicly available. They are clearly identified in Table 3 and the reference list.

Aguinis H, Ramani RS, Alabduljader N (2020) Best-practice recommendations for producers, evaluators, and users of methodological literature reviews. Organ Res Methods. https://doi.org/10.1177/1094428120943281

Article   Google Scholar  

Beske P, Land A, Seuring S (2014) Sustainable supply chain management practices and dynamic capabilities in the food industry: a critical analysis of the literature. Int J Prod Econ 152:131–143. https://doi.org/10.1016/j.ijpe.2013.12.026

Brandenburg M, Rebs T (2015) Sustainable supply chain management: a modeling perspective. Ann Oper Res 229:213–252. https://doi.org/10.1007/s10479-015-1853-1

Carter CR, Rogers DS (2008) A framework of sustainable supply chain management: moving toward new theory. Int Jnl Phys Dist Logist Manage 38:360–387. https://doi.org/10.1108/09600030810882816

Carter CR, Washispack S (2018) Mapping the path forward for sustainable supply chain management: a review of reviews. J Bus Logist 39:242–247. https://doi.org/10.1111/jbl.12196

Clark WR, Clark LA, Raffo DM, Williams RI (2021) Extending fisch and block’s (2018) tips for a systematic review in management and business literature. Manag Rev Q 71:215–231. https://doi.org/10.1007/s11301-020-00184-8

Crane A, Henriques I, Husted BW, Matten D (2016) What constitutes a theoretical contribution in the business and society field? Bus Soc 55:783–791. https://doi.org/10.1177/0007650316651343

Davis J, Mengersen K, Bennett S, Mazerolle L (2014) Viewing systematic reviews and meta-analysis in social research through different lenses. Springerplus 3:511. https://doi.org/10.1186/2193-1801-3-511

Davis HTO, Crombie IK (2001) What is asystematicreview? http://vivrolfe.com/ProfDoc/Assets/Davis%20What%20is%20a%20systematic%20review.pdf . Accessed 22 February 2019

De Lima FA, Seuring S, Sauer PC (2021) A systematic literature review exploring uncertainty management and sustainability outcomes in circular supply chains. Int J Prod Res. https://doi.org/10.1080/00207543.2021.1976859

Denyer D, Tranfield D (2009) Producing a systematic review. In: Buchanan DA, Bryman A (eds) The Sage handbook of organizational research methods. Sage Publications Ltd, Thousand Oaks, CA, pp 671–689

Google Scholar  

Devece C, Ribeiro-Soriano DE, Palacios-Marqués D (2019) Coopetition as the new trend in inter-firm alliances: literature review and research patterns. Rev Manag Sci 13:207–226. https://doi.org/10.1007/s11846-017-0245-0

Dieste M, Sauer PC, Orzes G (2022) Organizational tensions in industry 4.0 implementation: a paradox theory approach. Int J Prod Econ 251:108532. https://doi.org/10.1016/j.ijpe.2022.108532

Donthu N, Kumar S, Mukherjee D, Pandey N, Lim WM (2021) How to conduct a bibliometric analysis: an overview and guidelines. J Bus Res 133:285–296. https://doi.org/10.1016/j.jbusres.2021.04.070

Durach CF, Kembro J, Wieland A (2017) A new paradigm for systematic literature reviews in supply chain management. J Supply Chain Manag 53:67–85. https://doi.org/10.1111/jscm.12145

Fink A (2010) Conducting research literature reviews: from the internet to paper, 3rd edn. SAGE, Los Angeles

Fisch C, Block J (2018) Six tips for your (systematic) literature review in business and management research. Manag Rev Q 68:103–106. https://doi.org/10.1007/s11301-018-0142-x

Fritz MMC, Silva ME (2018) Exploring supply chain sustainability research in Latin America. Int Jnl Phys Dist Logist Manag 48:818–841. https://doi.org/10.1108/IJPDLM-01-2017-0023

Garcia-Torres S, Albareda L, Rey-Garcia M, Seuring S (2019) Traceability for sustainability: literature review and conceptual framework. Supp Chain Manag 24:85–106. https://doi.org/10.1108/SCM-04-2018-0152

Hanelt A, Bohnsack R, Marz D, Antunes Marante C (2021) A systematic review of the literature on digital transformation: insights and implications for strategy and organizational change. J Manag Stud 58:1159–1197. https://doi.org/10.1111/joms.12639

Kache F, Seuring S (2014) Linking collaboration and integration to risk and performance in supply chains via a review of literature reviews. Supp Chain Mnagmnt 19:664–682. https://doi.org/10.1108/SCM-12-2013-0478

Khalid RU, Seuring S (2019) Analyzing base-of-the-pyramid research from a (sustainable) supply chain perspective. J Bus Ethics 155:663–686. https://doi.org/10.1007/s10551-017-3474-x

Koufteros X, Mackelprang A, Hazen B, Huo B (2018) Structured literature reviews on strategic issues in SCM and logistics: part 2. Int Jnl Phys Dist Logist Manage 48:742–744. https://doi.org/10.1108/IJPDLM-09-2018-363

Kraus S, Breier M, Dasí-Rodríguez S (2020) The art of crafting a systematic literature review in entrepreneurship research. Int Entrep Manag J 16:1023–1042. https://doi.org/10.1007/s11365-020-00635-4

Kraus S, Mahto RV, Walsh ST (2021) The importance of literature reviews in small business and entrepreneurship research. J Small Bus Manag. https://doi.org/10.1080/00472778.2021.1955128

Kraus S, Breier M, Lim WM, Dabić M, Kumar S, Kanbach D, Mukherjee D, Corvello V, Piñeiro-Chousa J, Liguori E, Palacios-Marqués D, Schiavone F, Ferraris A, Fernandes C, Ferreira JJ (2022) Literature reviews as independent studies: guidelines for academic practice. Rev Manag Sci 16:2577–2595. https://doi.org/10.1007/s11846-022-00588-8

Leuschner R, Rogers DS, Charvet FF (2013) A meta-analysis of supply chain integration and firm performance. J Supply Chain Manag 49:34–57. https://doi.org/10.1111/jscm.12013

Lim WM, Rasul T (2022) Customer engagement and social media: revisiting the past to inform the future. J Bus Res 148:325–342. https://doi.org/10.1016/j.jbusres.2022.04.068

Lim WM, Yap S-F, Makkar M (2021) Home sharing in marketing and tourism at a tipping point: what do we know, how do we know, and where should we be heading? J Bus Res 122:534–566. https://doi.org/10.1016/j.jbusres.2020.08.051

Lim WM, Kumar S, Ali F (2022) Advancing knowledge through literature reviews: ‘what’, ‘why’, and ‘how to contribute.’ Serv Ind J 42:481–513. https://doi.org/10.1080/02642069.2022.2047941

Lusiantoro L, Yates N, Mena C, Varga L (2018) A refined framework of information sharing in perishable product supply chains. Int J Phys Distrib Logist Manag 48:254–283. https://doi.org/10.1108/IJPDLM-08-2017-0250

Maestrini V, Luzzini D, Maccarrone P, Caniato F (2017) Supply chain performance measurement systems: a systematic review and research agenda. Int J Prod Econ 183:299–315. https://doi.org/10.1016/j.ijpe.2016.11.005

Miemczyk J, Johnsen TE, Macquet M (2012) Sustainable purchasing and supply management: a structured literature review of definitions and measures at the dyad, chain and network levels. Supp Chain Mnagmnt 17:478–496. https://doi.org/10.1108/13598541211258564

Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6:e1000097. https://doi.org/10.1371/journal.pmed.1000097

Mukherjee D, Lim WM, Kumar S, Donthu N (2022) Guidelines for advancing theory and practice through bibliometric research. J Bus Res 148:101–115. https://doi.org/10.1016/j.jbusres.2022.04.042

Mulrow CD (1987) The medical review article: state of the science. Ann Intern Med 106:485–488. https://doi.org/10.7326/0003-4819-106-3-485

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. J Clin Epidemiol 134:178–189. https://doi.org/10.1016/j.jclinepi.2021.03.001

Pagell M, Wu Z (2009) Building a more complete theory of sustainable supply chain management using case studies of 10 exemplars. J Supply Chain Manag 45:37–56. https://doi.org/10.1111/j.1745-493X.2009.03162.x

Paul J, Criado AR (2020) The art of writing literature review: What do we know and what do we need to know? Int Bus Rev 29:101717. https://doi.org/10.1016/j.ibusrev.2020.101717

Paul J, Lim WM, O’Cass A, Hao AW, Bresciani S (2021) Scientific procedures and rationales for systematic literature reviews (SPAR-4-SLR). Int J Consum Stud. https://doi.org/10.1111/ijcs.12695

Pearce JM (2018) How to perform a literature review with free and open source software. Pract Assess Res Eval 23:1–13

Rhaiem K, Amara N (2021) Learning from innovation failures: a systematic review of the literature and research agenda. Rev Manag Sci 15:189–234. https://doi.org/10.1007/s11846-019-00339-2

Rojas-Córdova C, Williamson AJ, Pertuze JA, Calvo G (2022) Why one strategy does not fit all: a systematic review on exploration–exploitation in different organizational archetypes. Rev Manag Sci. https://doi.org/10.1007/s11846-022-00577-x

Sauer PC (2021) The complementing role of sustainability standards in managing international and multi-tiered mineral supply chains. Resour Conserv Recycl 174:105747. https://doi.org/10.1016/j.resconrec.2021.105747

Sauer PC, Seuring S (2017) Sustainable supply chain management for minerals. J Clean Prod 151:235–249. https://doi.org/10.1016/j.jclepro.2017.03.049

Seuring S, Gold S (2012) Conducting content-analysis based literature reviews in supply chain management. Supp Chain Mnagmnt 17:544–555. https://doi.org/10.1108/13598541211258609

Seuring S, Müller M (2008) From a literature review to a conceptual framework for sustainable supply chain management. J Clean Prod 16:1699–1710. https://doi.org/10.1016/j.jclepro.2008.04.020

Seuring S, Yawar SA, Land A, Khalid RU, Sauer PC (2021) The application of theory in literature reviews: illustrated with examples from supply chain management. Int J Oper Prod Manag 41:1–20. https://doi.org/10.1108/IJOPM-04-2020-0247

Siems E, Land A, Seuring S (2021) Dynamic capabilities in sustainable supply chain management: an inter-temporal comparison of the food and automotive industries. Int J Prod Econ 236:108128. https://doi.org/10.1016/j.ijpe.2021.108128

Snyder H (2019) Literature review as a research methodology: an overview and guidelines. J Bus Res 104:333–339. https://doi.org/10.1016/j.jbusres.2019.07.039

Spens KM, Kovács G (2006) A content analysis of research approaches in logistics research. Int Jnl Phys Dist Logist Manage 36:374–390. https://doi.org/10.1108/09600030610676259

Tachizawa EM, Wong CY (2014) Towards a theory of multi-tier sustainable supply chains: a systematic literature review. Supp Chain Mnagmnt 19:643–663. https://doi.org/10.1108/SCM-02-2014-0070

Tipu SAA (2022) Organizational change for environmental, social, and financial sustainability: a systematic literature review. Rev Manag Sci 16:1697–1742. https://doi.org/10.1007/s11846-021-00494-5

Touboulic A, Walker H (2015) Theories in sustainable supply chain management: a structured literature review. Int Jnl Phys Dist Logist Manage 45:16–42. https://doi.org/10.1108/IJPDLM-05-2013-0106

Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag 14:207–222. https://doi.org/10.1111/1467-8551.00375

Tröster R, Hiete M (2018) Success of voluntary sustainability certification schemes: a comprehensive review. J Clean Prod 196:1034–1043. https://doi.org/10.1016/j.jclepro.2018.05.240

Wang Y, Han JH, Beynon-Davies P (2019) Understanding blockchain technology for future supply chains: a systematic literature review and research agenda. Supp Chain Mnagmnt 24:62–84. https://doi.org/10.1108/SCM-03-2018-0148

Webster J, Watson RT (2002) Analyzing the past to prepare for the future: writing a literature review. MIS Q 26:xiii–xxiii

Wiese A, Kellner J, Lietke B, Toporowski W, Zielke S (2012) Sustainability in retailing: a summative content analysis. Int J Retail Distrib Manag 40:318–335. https://doi.org/10.1108/09590551211211792

Xiao Y, Watson M (2019) Guidance on conducting a systematic literature review. J Plan Educ Res 39:93–112. https://doi.org/10.1177/0739456X17723971

Yavaprabhas K, Pournader M, Seuring S (2022) Blockchain as the “trust-building machine” for supply chain management. Ann Oper Res. https://doi.org/10.1007/s10479-022-04868-0

Zhu Q, Bai C, Sarkis J (2022) Blockchain technology and supply chains: the paradox of the atheoretical research discourse. Transp Res Part E Logist Transp Rev 164:102824. https://doi.org/10.1016/j.tre.2022.102824

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

EM Strasbourg Business School, Université de Strasbourg, HuManiS UR 7308, 67000, Strasbourg, France

Philipp C. Sauer

Chair of Supply Chain Management, Faculty of Economics and Management, The University of Kassel, Kassel, Germany

Stefan Seuring

You can also search for this author in PubMed   Google Scholar

Contributions

The article is based on the idea and extensive experience of SS. The literature search and data analysis has mainly been performed by PCS and supported by SS before the paper manuscript has been written and revised in a common effort of both authors.

Corresponding author

Correspondence to Stefan Seuring .

Ethics declarations

Conflict of interest.

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Sauer, P.C., Seuring, S. How to conduct systematic literature reviews in management research: a guide in 6 steps and 14 decisions. Rev Manag Sci 17 , 1899–1933 (2023). https://doi.org/10.1007/s11846-023-00668-3

Download citation

Received : 29 September 2022

Accepted : 17 April 2023

Published : 12 May 2023

Issue Date : July 2023

DOI : https://doi.org/10.1007/s11846-023-00668-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Methodology
• Replicability
• Research process
• Structured literature review
• Systematic literature review

JEL Classification

• Find a journal
• Publish with us
• Track your research

Literature Reviews

• What is a Literature Review?
• Steps for Creating a Literature Review
• Providing Evidence / Critical Analysis
• Challenges when writing a Literature Review
• Systematic Literature Reviews

Steps for Cr eating Literature Reviews

• Formulating Research Questions and Objectives : Define the scope and focus of the review.
• Searching the Literature : Conduct a thorough search of academic databases and other sources.
• Screening for Inclusion : Select relevant studies based on predefined criteria.
• Assessing Quality : Evaluate the quality and reliability of the selected studies.
• Extracting Data : Collect relevant data from the studies.
• Analyzing Data : Synthesize and interpret the data to draw conclusions.
• << Previous: What is a Literature Review?
• Next: Developing a Literature Review >>
• Last Updated: Sep 4, 2024 11:43 AM
• URL: https://library.lsbu.ac.uk/literaturereviews

• Search Menu
• Sign in through your institution
• Advance articles
• Editor's Choice
• 100 years of the AJE
• Collections
• Author Guidelines
• Submission Site
• Open Access Options
• About American Journal of Epidemiology
• About the Johns Hopkins Bloomberg School of Public Health
• Journals Career Network
• Editorial Board
• Advertising and Corporate Services
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Assessing the certainty of the evidence in systematic reviews: importance, process, and use.

• Article contents
• Figures & tables
• Supplementary Data

Romina Brignardello-Petersen, Gordon H Guyatt, Assessing the Certainty of the Evidence in Systematic Reviews: Importance, Process, and Use, American Journal of Epidemiology , 2024;, kwae332, https://doi.org/10.1093/aje/kwae332

• Permissions Icon Permissions

When interpreting results and drawing conclusions, authors of systematic reviews should consider the limitations of the evidence included in their review. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach provides a framework for the explicit consideration of the limitations of the evidence included in a systematic review, and for incorporating this assessment into the conclusions. Assessments of certainty of evidence are a methodological expectation of systematic reviews. The certainty of the evidence is specific to each outcome in a systematic review, and can be rated as high, moderate, low, or very low. Because it will have an important impact, before conducting certainty of evidence, reviewers must clarify the intent of their question: are they interested in causation or association. Serious concerns regarding limitations in the study design, inconsistency, imprecision, indirectness, and publication bias can decrease the certainty of the evidence. Using an example, this article describes and illustrates the importance and the steps for assessing the certainty of evidence and drawing accurate conclusions in a systematic review.

• publication bias
• grade approach

Email alerts

Citing articles via, looking for your next opportunity.

• Recommend to your Library

Affiliations

• Online ISSN 1476-6256
• Print ISSN 0002-9262
• Copyright © 2024 Johns Hopkins Bloomberg School of Public Health
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

How to write a literature review introduction (+ examples)

The introduction to a literature review serves as your reader’s guide through your academic work and thought process. Explore the significance of literature review introductions in review papers, academic papers, essays, theses, and dissertations. We delve into the purpose and necessity of these introductions, explore the essential components of literature review introductions, and provide step-by-step guidance on how to craft your own, along with examples.

Why you need an introduction for a literature review

In academic writing , the introduction for a literature review is an indispensable component. Effective academic writing requires proper paragraph structuring to guide your reader through your argumentation. This includes providing an introduction to your literature review.

It is imperative to remember that you should never start sharing your findings abruptly. Even if there isn’t a dedicated introduction section .

When you need an introduction for a literature review

There are three main scenarios in which you need an introduction for a literature review:

What to include in a literature review introduction

It is crucial to customize the content and depth of your literature review introduction according to the specific format of your academic work.

In practical terms, this implies, for instance, that the introduction in an academic literature review paper, especially one derived from a systematic literature review , is quite comprehensive. Particularly compared to the rather brief one or two introductory sentences that are often found at the beginning of a literature review section in a standard academic paper. The introduction to the literature review chapter in a thesis or dissertation again adheres to different standards.

Academic literature review paper

The introduction of an academic literature review paper, which does not rely on empirical data, often necessitates a more extensive introduction than the brief literature review introductions typically found in empirical papers. It should encompass:

Regular literature review section in an academic article or essay

In a standard 8000-word journal article, the literature review section typically spans between 750 and 1250 words. The first few sentences or the first paragraph within this section often serve as an introduction. It should encompass:

In some cases, you might include:

Introduction to a literature review chapter in thesis or dissertation

Some students choose to incorporate a brief introductory section at the beginning of each chapter, including the literature review chapter. Alternatively, others opt to seamlessly integrate the introduction into the initial sentences of the literature review itself. Both approaches are acceptable, provided that you incorporate the following elements:

Examples of literature review introductions

Example 1: an effective introduction for an academic literature review paper.

To begin, let’s delve into the introduction of an academic literature review paper. We will examine the paper “How does culture influence innovation? A systematic literature review”, which was published in 2018 in the journal Management Decision.

Example 2: An effective introduction to a literature review section in an academic paper

The second example represents a typical academic paper, encompassing not only a literature review section but also empirical data, a case study, and other elements. We will closely examine the introduction to the literature review section in the paper “The environmentalism of the subalterns: a case study of environmental activism in Eastern Kurdistan/Rojhelat”, which was published in 2021 in the journal Local Environment.

The paper begins with a general introduction and then proceeds to the literature review, designated by the authors as their conceptual framework. Of particular interest is the first paragraph of this conceptual framework, comprising 142 words across five sentences:

Thus, the author successfully introduces the literature review, from which point onward it dives into the main concept (‘subalternity’) of the research, and reviews the literature on socio-economic justice and environmental degradation.

Examples 3-5: Effective introductions to literature review chapters

Numerous universities offer online repositories where you can access theses and dissertations from previous years, serving as valuable sources of reference. Many of these repositories, however, may require you to log in through your university account. Nevertheless, a few open-access repositories are accessible to anyone, such as the one by the University of Manchester . It’s important to note though that copyright restrictions apply to these resources, just as they would with published papers.

Master’s thesis literature review introduction

Phd thesis literature review chapter introduction, phd thesis literature review introduction.

The last example is the doctoral thesis Metacognitive strategies and beliefs: Child correlates and early experiences Chan, K. Y. M. (Author). 31 Dec 2020 . The author clearly conducted a systematic literature review, commencing the review section with a discussion of the methodology and approach employed in locating and analyzing the selected records.

Steps to write your own literature review introduction

Master academia, get new content delivered directly to your inbox, the best answers to "what are your plans for the future", 10 tips for engaging your audience in academic writing, related articles, 37 creative ways to get motivation to study, minimalist writing for a better thesis, separating your self-worth from your phd work, how to develop an awesome phd timeline step-by-step.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.8(11); 2016 Nov

How to Conduct a Systematic Review: A Narrative Literature Review

Nusrat jahan.

1 Psychiatry, Mount Sinai Chicago

Sadiq Naveed

2 Psychiatry, KVC Prairie Ridge Hospital

Muhammad Zeshan

3 Department of Psychiatry, Bronx Lebanon Hospital Icahn School of Medicine at Mount Sinai, Bronx, NY

Muhammad A Tahir

4 Psychiatry, Suny Upstate Medical University, Syracuse, NY

Systematic reviews are ranked very high in research and are considered the most valid form of medical evidence. They provide a complete summary of the current literature relevant to a research question and can be of immense use to medical professionals. Our goal with this paper is to conduct a narrative review of the literature about systematic reviews and outline the essential elements of a systematic review along with the limitations of such a review.

Introduction and background

A literature review provides an important insight into a particular scholarly topic. It compiles published research on a topic, surveys different sources of research, and critically examines these sources [ 1 ]. A literature review may be argumentative, integrative, historical, methodological, systematic, or theoretical, and these approaches may be adopted depending upon the types of analysis in a particular study [ 2 ].

Our topic of interest in this article is to understand the different steps of conducting a systematic review. Systematic reviews, according to Wright, et al., are defined as a “review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review” [ 3 ]. A systematic review provides an unbiased assessment of these studies [ 4 ]. Such reviews emerged in the 1970s in the field of social sciences. Systematic reviews, as well as the meta-analyses of the appropriate studies, can be the best form of evidence available to clinicians [ 3 ]. The unsystematic narrative review is more likely to include only research selected by the authors, which introduces bias and, therefore, frequently lags behind and contradicts the available evidence [ 5 ].

Epidemiologist Archie Cochrane played a vital role in formulating the methodology of the systematic review [ 6 ]. Dr. Cochrane loved to study patterns of disease and how these related to the environment. In the early 1970s, he found that many decisions in health care were made without reliable, up-to-date evidence about the treatments used [ 6 ].

A systematic review may or may not include meta-analysis, depending on whether results from different studies can be combined to provide a meaningful conclusion. David Sackett defined meta-analysis as a “specific statistical strategy for assembling the results of several studies into a single estimate” [ 7 - 8 ].

While the systematic review has several advantages, it has several limitations which can affect the conclusion. Inadequate literature searches and heterogeneous studies can lead to false conclusions. Similarly, the quality of assessment is an important step in systematic reviews, and it can lead to adverse consequences if not done properly.

The purpose of this article is to understand the important steps involved in conducting a systematic review of all kinds of clinical studies. We conducted a narrative review of the literature about systematic reviews with a special focus on articles that discuss conducting reviews of randomized controlled trials. We discuss key guidelines and important terminologies and present the advantages and limitations of systematic reviews.

Narrative reviews are a discussion of important topics on a theoretical point of view, and they are considered an important educational tool in continuing medical education [ 9 ]. Narrative reviews take a less formal approach than systematic reviews in that narrative reviews do not require the presentation of the more rigorous aspects characteristic of a systematic review such as reporting methodology, search terms, databases used, and inclusion and exclusion criteria [ 9 ]. With this in mind, our narrative review will give a detailed explanation of the important steps of a systematic review.

Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) checklist

Systematic reviews are conducted based on predefined criteria and protocol. The PRISMA-P checklist, developed by Moher, et al., contains 17 items (26 including sub-items) comprising the important steps of a systematic review, including information about authors, co-authors, their mailing and email addresses, affiliations, and any new or updated version of a previous systematic review [ 9 ]. It also identifies a plan for documenting important protocol amendments, registry names, registration numbers, financial disclosures, and other support services [ 10 ]. Moher, et al. also state that methods of systematic reviews involve developing eligibility criteria and describing information sources, search strategies, study selection processes, outcomes, assessment of bias in individual studies, and data synthesis [ 10 ].

Research question

Writing a research question is the first step in conducting a systematic review and is of paramount importance as it outlines both the need and validity of systematic reviews (Nguyen, et al., unpublished data). It also increases the efficiency of the review by limiting the time and cost of identifying and obtaining relevant literature [ 11 ]. The research question should summarize the main objective of a systematic review.

An example research question might read, “How does attention-deficit/hyperactivity disorder (ADHD) affect the academic performance of middle school children in North America?” The question focuses on the type of data, analysis, and topic to be discussed (i.e., ADHD among North American middle school students). Try to avoid research questions that are too narrow or broad—they can lead to the selection of only a few studies and the ability to generalize results to any other populations may be limited. An example of a research question that is too narrow would be, “What is the prevalence of ADHD in children and adolescents in Chicago, IL?” Alternately, if the research question is too broad, it can be difficult to reach a conclusion due to poor methodology. An example of a research question that is too broad in scope would be, “What are the effects of ADHD on the functioning of children and adolescents in North America?”

Different tools that can be used to help devise a research question, depending on the type of question, are: population, intervention, comparator, and outcomes (PICO); sample, phenomenon of interest, design, evaluation, and research type (SPIDER); setting, perspective, intervention, comparison, and evaluation (SPICE); and expectation, client group, location, impact, professionals, and service (ECLIPSE).

The PICO approach is mostly used to compare different interventions with each other. It helps to formulate a research question related to prognosis, diagnosis, and therapies [ 12 ].

Scenario: A 50-year-old white woman visited her psychiatrist with a diagnosis of major depressive disorder. She was prescribed fluoxetine, which she feels has been helpful. However, she experienced some unpleasant side effects of nausea and abdominal discomfort. She has recently been told by a friend about the use of St. John’s wort in treating depression and would like to try this in treating her current depression. (Formulating research questions, unpublished data).

In the above-mentioned scenario, the sample population is a 50-year-old female with major depressive disorder; the intervention is St. John’s wort; the comparison is fluoxetine; and the outcome would be efficacy and safety. In order to see the outcome of both efficacy and safety, we will compare the efficacy and safety of both St. John’s wort and fluoxetine in a sample population for treating depression. This scenario represents an example where we can apply the PICO approach to compare two interventions.

In contrast, the SPIDER approach is focused more on study design and samples rather than populations [ 13 ]. The SPIDER approach can be used in this research question: “What is the experience of psychiatry residents attending a transgender education?” The sample is psychiatry residents; the phenomenon of interest is transgender education; the design is a survey; the evaluation looks at the experience; and the research type is qualitative. 

The SPICE approach can be used to evaluate the outcome of a service, intervention, or project [ 14 ]. The SPICE approach applies to the following research question: “In psychiatry clinics, does the combined use of selective serotonin reuptake inhibitor (SSRI) and psychotherapy reduce depression in an outpatient clinic versus SSRI therapy alone?” The setting is the psychiatry clinic; the perspective/population is the outpatient; the intervention is combined psychotherapy and SSRI; the comparison is SSRI alone; and the evaluation is reduced depression. 

The ECLIPSE approach is useful for evaluating the outcome of a policy or service (Nguyen, et al., unpublished data). ECLIPSE can apply in the following research question: “How can a resident get access to medical records of patients admitted to inpatient from other hospitals?” The expectation is: “What are you looking to improve/change to increase access to medical records for patients admitted to inpatient?” The client group is the residents; the location is the inpatient setting; the impact would be the residents having easy access to medical records from other hospitals; and the professionals in this scenario would be those involved in improving the service experiences such as hospital administrators and IT staff.

Inclusion and exclusion criteria

Establishing inclusion and exclusion criteria come after formulating research questions. The concept of inclusion and exclusion of data in a systematic review provides a basis on which the reviewer draws valid and reliable conclusions regarding the effect of the intervention for the disorder under consideration [ 11 ]. Inclusions and exclusion are based on preset criteria for specific systematic review. It should be done before starting the literature search in order to minimize the possibility of bias.

Eligibility criteria provide the boundaries of the systematic review [ 15 ]. Participants, interventions, and comparison of a research question provide the basis for eligibility criteria [ 15 ]. The inclusion criteria should be able to identify the studies of interest and, if the inclusion criteria are too broad or too narrow, it can lead to an ineffective screening process.

Protocol registration

Developing and registering research protocol is another important step of conducting a systematic review. The research protocol ensures that a systematic review is carefully planned and explicitly documented before the review starts, thus promoting consistency in conduct for the review team and supporting the accountability, research integrity, and transparency of the eventually completed review [ 10 ]. PROSPERO and the Cochrane Database of Systematic Reviews are utilized for registering research protocols and research questions, and they check for prior existing duplicate protocols or research questions. PROSPERO is an international database of prospectively registered systematic reviews related to health care and social sciences (PRISMA, 2016). It is funded by the National Institute for Health Research. The Cochrane Collaboration concentrates on producing systematic reviews of interventions and diagnostic test accuracy but does not currently produce reviews on questions of prognosis or etiology [ 16 ].

A detailed and extensive search strategy is important for the systematic review since it minimizes bias in the review process [ 17 ].

Selecting and searching appropriate electronic databases is determined by the topic of interest. Important databases are: MEDLARS Online (MEDLINE), which is the online counterpart to the Medical Literature Analysis and Retrieval System (MEDLARS); Excerpta Medica Database (EMBASE); and Google Scholar. There are multiple electronic databases available based on the area of interest. Other important databases include: PsycINFO for psychology and psychiatry; Allied and Complementary Medicine Database (AMED) for complementary medicine; Manual, Alternative, and Natural Therapy Index System (MANTIS) for alternative medical literature; and Cumulative Index to Nursing and Allied Health Literature (CINAHL) for nursing and allied health [ 15 ].

Additional studies relevant for the review may be found by looking at the references of studies identified by different databases [ 15 ]. Non-indexed articles may be found by searching the content of journals, conferences proceedings, and abstracts. It will also help with letters and commentaries which may not get indexed [ 15 ]. Reviewing clinical trial registries can provide information about any ongoing trials or unpublished research [ 15 ]. A gray literature search can access unpublished papers, reports, and conference reports, and it generally covers studies that are published in an informal fashion, rather than in an indexed journal [ 15 ]. Further search can be performed by selecting important key articles and going through in-text citations [ 15 ].

Using Boolean operators, truncation, and wildcards

Boolean operators use the relationship between different search words to help with the search strategy. These are simple words (i.e., AND, OR, and NOT) which can help with more focused and productive results (poster, Jahan, et al.: How to conduct a systematic review. APPNA 39th Summer Convention. Washington, DC. 2016). The Boolean operator AND finds articles with all the search words. The use of OR broadens the focus of the search, and it will include articles with at least one search term. The researchers can also ignore certain results from the records by using NOT in the search strategy.

An example of AND would be using “depression” AND “children” in the search strategy with the goal of studying depression in children. This search strategy will include all the articles about both depression and children. The researchers may use OR if the emphasis of the study is mood disorders or affective disorders in adolescents. In that case, the search strategy will be “mood disorders” OR “affective disorders” AND “adolescents.” This search will find all the articles about mood disorders or affective disorders in adolescents. The researchers can use NOT if they only want to study depression in children and want to ignore bipolar disorder from the search. An example search in this scenario would be “depression” NOT “bipolar disorder” AND “children.” This will help ignore studies related to bipolar disorder in children.

Truncation and wildcards are other tools to make search strategy more comprehensive and focused. While the researchers search a database for certain articles, they frequently face terminologies that have the same initial root of a word but different endings. An example would be "autism," "autistic," and "autism spectrum disorder." These words have a similar initial root derived from “autis” but they end differently in each case. The truncation symbol (*) retrieves articles that contain words beginning with “autis” plus any additional characters. Wildcards are used for words with the same meanings but different spellings due to various reasons. For the words with spelling variations of a single letter, wildcard symbols can be used. When the researcher inputs “M+N” in the search bar, this returns results containing both “man” or “men” as the wildcard accounts for the spelling variations between the letters M and N.

Study selection

Study selection should be performed in a systematic manner, so reviewers deal with fewer errors and a lower risk of bias (online course, Li T, Dickersin K: Introduction to systematic review and meta-analysis. 2016. https://www.coursera.org/learn/systematic-review #). Study selection should involve two independent reviewers who select studies using inclusion and exclusion criteria. Any disagreements during this process should be resolved by discussion or by a third reviewer [ 10 ]. Specific study types can be selected depending on the research question. For example, questions on incidence and prevalence can be answered by surveys and cohort studies. Clinical trials can provide answers to questions related to therapy and screening. Queries regarding diagnostic accuracy can be answered by clinical trials and cross-sectional studies (online course, Li T, Dickersin K: Introduction to systematic review and meta-analysis. 2016. https://www.coursera.org/learn/systematic-review #). Prognosis and harm-related questions should use cohort studies and clinical trials, and etiology questions should use case-control and cohort studies (online course, Li T, Dickersin K: Introduction to systematic review and meta-analysis. 2016. https://www.coursera.org/learn/systematic-review #).

Data screening and data extractions are two of the major steps in conducting a systematic review [ 18 ]. Data screening involves searching for relevant articles in different databases using keywords. The next step of data screening is manuscript selection by reviewing each manuscript in the search results to compare that manuscript against the inclusion criteria [ 18 ]. The researchers should also review the references of the papers selected before selecting the final paper, which is the last step of data screening [ 18 ].

The next stage is extracting and appraising the data of the included articles [ 18 ]. A data extraction form should be used to help reduce the number of errors, and more than one person should record the data [ 17 ]. Data should be collected on specific points like population type, study authors, agency, study design, humanitarian crisis, target age groups, research strengths from the literature, setting, study country, type(s) of public health intervention, and health outcome(s) addressed by the public health intervention. All this information should then be put into an electronic database [ 18 ].

Assessing bias

Bias is a systematic error (or deviation from the truth) in results or inferences. Biases can change the results of any study and lead to an underestimation or overestimation of the true intervention effect [ 19 ]. Biases can impact any aspect of a review, including selecting studies, collecting and extracting data, and making a conclusion. Biases can vary in magnitude; some are small, with negligible effect, but some are substantial to a degree where an apparent finding may be entirely due to bias [ 19 ]. There are different types of bias, including, but not limited to, selection, detection, attrition, reporting, and performance.

Selection bias occurs when a sample selected is not representative of the whole general population. If randomization of the sample is done correctly, then chances of selection bias can be minimized [ 20 ].

Detection bias refers to systematic differences between groups in how outcomes are determined. This type of bias is based on knowledge of the intervention provided and its outcome [ 19 ].

Attrition bias refers to systematic differences between groups in withdrawals from a study [ 19 ]. The data will be considered incomplete if some subjects are withdrawn or have irregular visits during data collection.

Reporting bias refers to systematic differences between reported and unreported findings, and it is commonly seen during article reviews. Reporting bias is based on reviewer judgment about the outcome of selected articles [ 20 ].

Performance bias develops due to the knowledge of the allocated interventions by participants and personnel during the study [ 20 ]. Using a double-blind study design helps prevent performance bias, where neither the experimenter nor the subjects know which group contains controls and which group contains the test article [ 14 ].

Last step of systematic review: discussion

The discussion of a systematic review is where a summary of the available evidence for different outcomes is written and discussed [ 10 ]. The limitations of a systematic review are also discussed in detail. Finally, a conclusion is drawn after evaluating the results and considering limitations [ 10 ].

Discussion of the current article

Systematic reviews with or without a meta-analysis are currently ranked to be the best available evidence in the hierarchy of evidence-based practice [ 21 ]. We have discussed the methodology of a systematic review. A systematic review is classified in the category of filtered information because it appraises the quality of the study and its application in the field of medicine [ 21 ]. However, there are some limitations of the systematic review, as we mentioned earlier in our article. A large randomized controlled trial may provide a better conclusion than a systematic review of many smaller trials due to their larger sample sizes [ 22 ], which help the researchers generalize their conclusions for a bigger population. Other important factors to consider include higher dropout rates in large studies, co-interventions, and heterogeneity among studies included in the review.

As we discussed the limitations of the systematic review and its effect on quality of evidence, there are several tools to rate the evidence, such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system [ 22 ]. GRADE provides a structured approach to evaluating the risk of bias, serious inconsistency between studies, indirectness, imprecision of the results, and publication bias [ 22 ]. Another approach used to rate the quality of evidence is a measurement tool to assess systematic reviews (AMSTAR) [ 23 ]. It is also available in several languages [ 23 ].

Conclusions

Despite its limitations, a systematic review can add to the knowledge of the scientific community especially when there are gaps in the existing knowledge. However, conducting a systematic review requires different steps that involve different tools and strategies. It can be difficult at times to access and utilize these resources. A researcher can understand and strategize a systematic review following the different steps outlined in this literature review. However, conducting a systematic review requires a thorough understanding of all the concepts and tools involved, which is an extensive endeavor to be summed up in one article.

The Cochrane Handbook for Systematic Reviews of Interventions and the Center for Reviews and Dissemination (CRD) provide excellent guidance through their insightful and detailed guidelines. We recommend consulting these resources for further guidance.

Given that our article is a narrative review of the scholarly literature, it contains the same limitations as noted for any narrative review. We hope that our review of the means and methods for conducting a systematic review will be helpful in providing basic knowledge to utilize the resources available to the scientific community.

The content published in Cureus is the result of clinical experience and/or research by independent individuals or organizations. Cureus is not responsible for the scientific accuracy or reliability of data or conclusions published herein. All content published within Cureus is intended only for educational, research and reference purposes. Additionally, articles published within Cureus should not be deemed a suitable substitute for the advice of a qualified health care professional. Do not disregard or avoid professional medical advice due to content published within Cureus.

The authors have declared that no competing interests exist.

We Trust in Human Precision

20,000+ Professional Language Experts Ready to Help. Expertise in a variety of Niches.

API Solutions

• API Pricing
• Cost estimate
• Customer loyalty program
• Educational Discount
• Non-Profit Discount
• Green Initiative Discount1

Value-Driven Pricing

Unmatched expertise at affordable rates tailored for your needs. Our services empower you to boost your productivity.

• Special Discounts
• Enterprise transcription solutions
• Enterprise translation solutions
• Transcription/Caption API
• AI Transcription Proofreading API

Trusted by Global Leaders

GoTranscript is the chosen service for top media organizations, universities, and Fortune 50 companies.

GoTranscript

One of the Largest Online Transcription and Translation Agencies in the World. Founded in 2005.

Speaker 1: The first step of doing a systematic literature review is coming up with a review question, like what do you actually want to know about the world and how can you phrase that as a simple question. You can write down all of the questions you want and then choose from the best one or a combination but I like to go to ChatGPT and use them as like a sounding board and a research assistant so that they can help me really sort of refine what I actually want to do a systematic literature review on. So here we are, we head over and we say, help me define a systematic literature review research question about beards and their smell. Maybe that's what I was interested in. My beard smells lovely. It smells like Australian sandalwood at the moment. Beautiful. It says a systematic literature review research question should be specific blah blah blah. And then it comes up with one. How do microbial communities in beards influence blah blah blah. And it gives me kind of a first start. The one thing I found about any AI that you're asking, it makes a lot of assumptions about what you want to know. So I highly recommend that you go in and you sort of like re-prompt it and you say, I like this bit, but I don't like this bit, or this bit's good, but you're a little bit off on this area. That is how you kind of use this as a research assistant as like a sounding board for all of your ideas. Then once you've got a research question and you need to spend probably most of the time of the first bit of searching on this because it's so very important. Come up with a definitive but broad, and I know that is so contradictory, but you need to come up with something that is focused enough that it will give you sort of like a good outcome but not too broad that all of a sudden, you know, like you're dealing with thousands and thousands of papers. So that is the challenge, and use ChatGPT to get that balance. Now, you can also use frameworks. There's different frameworks that you can use which will help you with this first sort of like step. And I just asked ChatGPT. I'm familiar with some of these, but some of these were new to me as well. I said, what frameworks for a systematic literature review can be used for this question? And it says Prisma, it used Cochrane Handbook for systematic reviews, it's got the Joanna Briggs Institute Methodology, Spyder and Pico. One of the most famous ones arguably is Pico where you say, okay, I've got this P, population, I've got this I, intervention that I'm looking at, I've got this C, comparison of all of the things that I found and O, outcome. Then what happened when they did these things? And quite often the C stands for comparison because it's a quantitative measurement of comparing it to say like a placebo if you're doing a lot of health stuff or another sort of intervention. So that's how we use frameworks to start thinking about our research question. What population are we gonna look at? What intervention are we looking at? What comparison, if any, are we gonna look at? And we're gonna look for the outcomes within those systems and structures that we set in place. So that's step one. Step two, actually, is what defines a literature review from a systematic literature review? Let's get into that. This is so very important for a systematic literature review because we need to know what methods we are going to use to filter all of the different stuff that we're gonna come across. We wanna know stuff like what procedure are we gonna go through to find the literature. We wanna know what keywords we're gonna use, what semantic search terms we're gonna use in certain databases to find the literature. Now, I like to head over to something like Search Smart. This will give you sort of like the best databases to search for your systematic literature review. And so all you need to do is look for scholarly records or clinical trials if you want, put in the subjects or the keywords and then sort of like define whether or not you want systemic keyword searching, backwards citation, forwards, all of that sort of stuff and also non-paywall databases and you click Start Comparison and it will go off and give you all of the different databases that you can look at. Then, keywords. Keywords are so very important because we often find research based on how they're described like in the abstract or the title. So be very specific with your keywords. By the way, I have another video, go check it out here, where I talk about how to find all of the literature that you'll ever need using different approaches, AI, Boolean searches, old school keyword searches, and that video will allow you to find everything you need in your systematic review. But databases are very important. Where are you gonna search? what keywords are you gonna search for, what semantic search questions, and that's new for this sort of like era of AI because it allows us to actually just put our research question into a database and have it sort of understand that question and give us results back. So now we're on to the exciting part which is finding the research papers. The one thing I like to do first and foremost, and that's only possible now because of AI's semantic search. I love it so much. Let's head over to the three tools that I think you would wanna use. The first one is Elicit. Ask a research question. Beards and, ooh, not bears, and smells. Let's see, that's not really a research question, but let's see what it comes up with. But it's that sort of stuff that you need to sort of like thinking about. Like, is that a keyword combination that you want to put in all of the databases or not? Whatever you decide using your meat brain. So, here we go. Here's all of the different papers that I could talk about. Brilliant. The next one is consensus. Beards and smell. Then we can go off and find all of the papers here using that sort of semantic keyword search as well. And we've also got size space. I can go here, beards and smell. And this is where I like to find all of my stuff using keywords and semantic search. So making sense, oh, this hasn't really done too well with beards, beards and issues, blah, blah, blah. So overall, you can see that we've got a little bit of discrepancy between what these pick up. So it's very important, I think, that you try a few to see what works best for you. And then finally, we gotta head over to something like Google Scholar, and we wanna say, okay, what keywords are we gonna put in? This isn't semantic search, this is just putting in beards and smell. And we can use Boolean operators to make sure that we're actually gonna get the papers that are relevant for us. So we can go beards, and then and, because we want and, smell. There we are. So then we're gonna come up with all of the smell and beard articles that it's going to come up with. The smell report, shame and glory. Only the beards, even after beards became merely rather than daring, the rather radical, oh my God, I don't like this one. The British Journal of Sociology, come on now, you can do better than that. But that is where you can go and actually find all of this information. And so semantic, keywords, databases, and Boolean operators to have a look at what you're excluding and including in your search is very, very important. So that is the step three. Yeah, step three, that is searching for the paper. And now we need to filter and screen and read. Once we've ended up with a load of papers from our searching based on the criteria and the methods we set out in step two, we've now got like an exclusion and inclusion protocol where we need to say, okay, we've got all of these studies, Which ones are we going to include and which ones are we going to exclude? And it's a really sort of like simple process of just filtering. This is why you need a load of papers at the top. Put loads of papers at the top and then they have to filter down to the useful papers down the bottom. And it may only be a small fraction of all of the papers you found, but this is what a systematic review is all about. It's about making sure that we include the papers that are relevant for your research question and not just like general themes, which is like a normal literature review where we just sort of say, oh yeah, there's this theme and this theme and this theme. No, this one's much more focused, so we need to filter it. I like to use the Prisma flowchart to work out which ones I'm getting rid of and keep track of the ones I've got rid of and how much I've filtered it down. So a Prisma flowchart looks like this. We've got identification in the top here and then we've got records identified through database searching. In this case, they had 96. and then we've got other additional identified through other sources, and this was none in this bit. Then they removed duplicates, so there was two that were the same, so they removed one of them, and then they said, okay, we've got this many in screen, 95, and eligibility, full text articles assessed for eligibility, there was only five, and all of these were actually excluded because it didn't meet their criteria that they'd set out in part of their exclusion or inclusion criteria. So you can see we've got like examines treatment, not prevention. So this was like obviously like a health study where they were looking at treatment and not the prevention or something. So that was most of them, that was 52. Then one was pediatric, one was irrelevant. Oh no, loads were irrelevant, 37 were irrelevant. So you can see we've gone from 96 all the way down to five at this point. And then full text articles not included. Well, there was none there, which is great. but here we've got four which studies included in quantitative synthesis or a meta-analysis was only four, they got rid of 92 of them because they didn't meet the specific search and exclusion and inclusion criteria that they set. That is so important and that is very, very typical of a systematic review. So now it's about taking those special studies that you found and getting all of the important stuff out of them. you should read them, especially if there's only four. You should read them from end to beginning. No, don't read them like that. Read them however you want, normally with abstract, then to conclusions, then to introduction, then to method, anyway, you get the idea. Do you know what, actually, I've got another video on how to read like a PhD. Go check out that one there. It's much better than what I just said. But now you need to read them and you need to start thinking about how these studies are influencing your research question sort of response. Are they for it? Are they against it? Do they give you a new insight? Is there something sneaky in there when you look at them all together that is surprising? It's those sort of things that really should be sort of milling around in your head. We're not looking for any sort of definitive stuff just yet, but we just need to read, analyze, refine, understand, all of those stuff. Those words are very important, put them there. But now, we've got a couple of new ways that we can actually talk to all of our documents. So one place I really like is docanalyzer.ai and what you can do is upload your documents and tag them as, in this case I've got literature review, you can see I've got one, two, three, four, five, six here. So then we can go to labels and we can go chat with these six documents. And the one thing I love about docanalyzer is that it doesn't like try to make stuff up. If it doesn't understand what you're asking or it can't identify it in the documents that you've given it, it will just say, hey, I don't really know, can you give me a bit more information? It doesn't sort of like BS its way into chat, which I really like. So, for example here, it says to identify the important parts of the document, I would need more specific keywords or topics of interest. That's what I want from an AI, something that isn't just gonna make stuff up. Another thing you can do is head back over to size space, And in SciSpace, you can actually get results from my library. So if you put those very specific studies that you've filtered and found into your library, you can then ask it questions across that library, which I think is really, really fantastic. So not only do you read it all, if you can, if it's a sensible amount of papers, but then you can start chatting to all of the documents together in something like DocAnalyzer and SciSpace, and then you can get sort of further connections, further deeper inquiry into things that maybe you have missed. Or maybe there's just a question, you've read them all, and there's a question sort of in your mind. You're like, actually, does this apply to all of the papers or not? Put it into something like this and it will search across all of your documents. I absolutely love, I'm doing this today, Chef's Kiss, it's my new favorite thing. Chef's Kiss, yum, yum, yum, yum, yum. But doing that means that you're not gonna miss out on anything because you're going to use old school tactics by just reading, read, read, read, read, read, and new school tactics by using AI, AI, AI, AI. Together, they are the perfect combination, yes. And then it's all about writing it up, making sure that you actually talk about what your research question is, the methods you've used, the filtration criteria, and the exclusion and inclusion criteria, the keywords you search for, then what you've found, how they all sort of like relate together, and the outcome. What is the outcome of this literature review? Does it support your research questions? Does it give you a new insight? That is how you write this. That is the structure. It is so very sort of systematic. A systematic literature review has to be systematic, otherwise you'll just end up being completely lost in all of the papers. Oh, so many papers, so many papers. Filter them out, find the good ones, write it out. Brilliant. All right, if you like this video, Go check out this one where I talk about how to write an exceptional literature review with AI. It's going to be a great sort of addition to what you've learned in here. Go check it out.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

The impact of speed limit change on emissions: a systematic review of literature.

1. Introduction

1.1. background, 2. materials and methods, 2.1. protocol and search strategy.

• Firstly, a comprehensive search was conducted using different bibliographic databases to retrieve published literature on the topic. These databases included Scopus, Google Scholar, ScienceDirect, and Research gates. A search query was created with the following limitations: published “year > 2000”; Document Type “Article or Review or Conference Paper”; Source Type “Journals”; and Language “English” or “French”. This broad search aimed to capture a wide range of potentially relevant literature.
• Secondly, a Google search was performed to identify websites with published reports or gray literature relevant to the study topic. This was supplemented by subject matter expert consultations to identify any additional publications that may have been missed in the initial search and are relevant for the review. This step helped to supplement the database search and ensure that the review included a more comprehensive set of relevant literature.
• In the final stage, the reference lists and bibliographies of the sources identified in the previous stages were thoroughly scrutinized to extract any additional relevant sources that may have been omitted in the previous stages. This supplementary reference list screening also helped to identify any relevant studies that were not indexed in the databases searched. The sources retrieved from all stages described above were considered relevant or irrelevant based on a pre-defined eligibility criterion (described in Section 2.1 ).

2.2. Eligibility Criteria

• Direct investigation of the impact of speed limit changes or mixed measures on vehicular emissions or pollutant emissions.
• Provision of quantitative or qualitative data on at least one of the following emissions or pollutants: nitrogen oxides (NO x ), carbon dioxide (CO 2 ), carbon monoxide (CO), and particulate matter (PM).
• Clear description of the study methodology or use of robust methodologies, including macro- and micro-modeling, simulation studies, on-road emission measurements, computational estimation, before-and-after evaluations, or a combination thereof.
• Publication in peer-reviewed journals, conference papers, or credible sources and websites to ensure the reliability of the findings.
• Studies in English or French, published after 2000.
• Excluded studies included studies with a lack of a clear comparative framework or absence of quantitative emission data/results; lack of full-text access or knowledge on methodologies employed for secondary sources; research not focusing on the direct relationship between real/simulated speed limits change and vehicular emissions; unclear methodologies for primary sources; and studies on emission rate and speed change models that are not focused on interventions (like speed limit change).

2.3. Screening, Documentation, and Analysis of Papers

3.1. general results of literature search, 3.2. review findings on the impact of speed limits interventions on emissions, 3.2.1. synthesis of studies in low-speed environments, 3.2.2. synthesis of studies in high-speed environment, 4. discussion, 5. conclusions and future implications, supplementary materials, author contributions, data availability statement, conflicts of interest.

• Fondzenyuy, S.K.; Jackai, I.N.; Feudjio, S.L.T.; Usami, D.S.; Gonzalez-Hernández, B.; Wounba, J.F.; Elambo, N.G.; Persia, L. Assessment of Sustainable Mobility Patterns of University Students: Case of Cameroon. Sustainability 2024 , 16 , 4591. [ Google Scholar ] [ CrossRef ]
• International Energy Agency CO 2 Emissions in 2023. 2023. Available online: https://iea.blob.core.windows.net/assets/33e2badc-b839-4c18-84ce-f6387b3c008f/CO2Emissionsin2023.pdf (accessed on 26 August 2024).
• EEA. Air Quality in Europe—2020 Report, no. No 09/2020. 2020. Available online: https://www.eea.europa.eu//publications/air-quality-in-europe-2020-report (accessed on 16 April 2024).
• Kunkler, J.; Braun, M.; Kellner, F. Speed limit induced CO 2 reduction on motorways: Enhancing discussion transparency through data enrichment of road networks. Sustainability 2021 , 13 , 395. [ Google Scholar ] [ CrossRef ]
• Baldasano, J.M.; Gonçalves, M.; Soret, A.; Jiménez-Guerrero, P. Air pollution impacts of speed limitation measures in large cities: The need for improving traffic data in a metropolitan area. Atmos. Environ. 2010 , 44 , 2997–3006. [ Google Scholar ] [ CrossRef ]
• World Health Organization. Managing Speed. 2017. Available online: https://www.who.int/publications/i/item/managing-speed (accessed on 20 May 2024).
• Ward, H.; Robertson, S.; Allsop, R. Managing speeds of traffic on European roads: Non-accident external and internal effects of vehicle use and how these depend on speed. In Proceedings of the Conference on Road Safety in Europe, Bergisch Ladbach, Germany, 21–23 September 1998; Volume 1. Available online: http://virtual.vtt.fi/master/pre12.pdf (accessed on 5 November 2023).
• Kamerud, D.B. Benefits and Costs of the 55 mph Speed Limit: New Estimates and Their Implications. J. Policy Anal. Manag. 1988 , 7 , 341–352. [ Google Scholar ] [ CrossRef ]
• Coelho, M.C.; Farias, T.L.; Rouphail, N.M. Impact of speed control traffic signals on pollutant emissions. Transp. Res. D Transp. Environ. 2005 , 10 , 323–340. [ Google Scholar ] [ CrossRef ]
• Cameron, M. Economic Analysis of Optimum Speeds on Rural State Highways in New Zealand, no. October 2012. 2022. Available online: https://www.nzta.govt.nz/assets/resources/economic-analysis-of-optimum-speeds-on-rural-state-highways-in-nz/Economic-analysis-of-optimum-speeds-on-rural-state-highways-in-nz.pdf (accessed on 5 November 2023).
• Boriboonsomsin, B.; Barth, M. Real-World Carbon Dioxide Impacts of Traffic Congestion. Transp. Res. Rec. 2008 , 2058 , 163–171. [ Google Scholar ]
• Van Benthem, A. What is the optimal speed limit on freeways? J. Public Econ. 2015 , 124 , 44–62. [ Google Scholar ] [ CrossRef ]
• Delhomme, P.; Chappé, J.; Grenier, K.; Pinto, M.; Martha, C. Reducing air-pollution: A new argument for getting drivers to abide by the speed limit? Accid. Anal. Prev. 2010 , 42 , 327–338. [ Google Scholar ] [ CrossRef ]
• Panis, L.I.; Broekx, S.; Liu, R. Modelling instantaneous traffic emission and the influence of traffic speed limits. Sci. Total Environ. 2006 , 371 , 270–285. [ Google Scholar ] [ CrossRef ]
• Panis, L.I.; Beckx, C.; Broekx, S.; De Vlieger, I.; Schrooten, L.; Degraeuwe, B.; Pelkmans, L. PM, NO x and CO 2 emission reductions from speed management policies in Europe. Transp. Policy 2011 , 18 , 32–37. [ Google Scholar ] [ CrossRef ]
• Alahmer, A. Performance and emission assessments for different acetone gasoline blends powered spark ignition engine. Int. J. Veh. Struct. Syst. 2018 , 10 , 127–132. [ Google Scholar ] [ CrossRef ]
• Thomaz, F.; Baeta, J.G.C. Effects of Engine Speed on the Performance at Extreme Vehicle Driving Conditions ; SAE Technical Paper; SAE International: Warrendale, PA, USA, 2020. [ Google Scholar ] [ CrossRef ]
• Bel, G.; Bolancé, C.; Guillén, M.; Rosell, J. The environmental effects of changing speed limits: A quantile regression approach. Transp. Res. D Transp. Environ. 2015 , 36 , 76–85. [ Google Scholar ] [ CrossRef ]
• Baltrėnas, P.; Januševičius, T.; Chlebnikovas, A. Research into the impact of speed bumps on particulate matter air pollution. Measurement 2017 , 100 , 62–67. [ Google Scholar ] [ CrossRef ]
• Perez-Prada, F.; Monzon, A. Ex-post environmental and traffic assessment of a speed reduction strategy in Madrid’s inner ring-road. J. Transp. Geogr. 2017 , 58 , 256–268. [ Google Scholar ] [ CrossRef ]
• Yannis, G.; Michelaraki, E. Review of City-Wide 30 km/h Speed Limit Benefits in Europe. Sustainability 2024 , 16 , 4382. [ Google Scholar ] [ CrossRef ]
• Ellison, A.B.; Greaves, S.P. Speeding in urban environments: Are the time savings worth the risk? Accid. Anal. Prev. 2015 , 85 , 239–247. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Default 20 mph Speed Limit on Restricted Roads Phase 1|Transport for Wales. Available online: https://tfw.wales/about-us/transparency/publications/default-20mph-speed-limit (accessed on 20 April 2023).
• Archer, J.; Fotheringham, N.; Symmons, M.; Corben, B. The Impact of Lowered Speed Limits in Urban Areas. 2006, pp. 1–7. Available online: https://acrs.org.au/files/arsrpe/RS07003.pdf (accessed on 30 October 2023).
• Dutschke, J.K.; Woolley, J.E. Simulation of Rural Travel Times to Quantify the Impact of Lower Speed Limits. Australasian Road Safety Research, Policing and Education Conference, 10–13 November 2009, Sydney, New South Wales. Available online: https://casr.adelaide.edu.au/casrpubfile/954/CASRsimulationtraveltimes1112.pdf (accessed on 29 October 2023).
• Cerema Lowering the Speed Limit to 80 km/h. Final Assessment Report. 2020. Available online: https://www.cerema.fr/en/actualites/lowering-maximum-authorised-speed-80-kmh-final-assessment (accessed on 7 October 2023).
• Mitra, S.; Job, S.; Han, S.; Eom, K. Do Speed Limit Reductions Help Road Safety? World Bank: Washington, DC, USA, 2021. [ Google Scholar ] [ CrossRef ]
• Xu, X.; Chen, A.; Lo, H.K.; Yang, C. Modeling the impacts of speed limits on uncertain road networks. Transp. A Transp. Sci. 2018 , 14 , 66–88. [ Google Scholar ] [ CrossRef ]
• Othman, B.; De Nunzio, G.; Di Domenico, D.; Canudas-De-Wit, C. Variable Speed Limits Control in an Urban Road Network to Reduce Environmental Impact of Traffic. In Proceedings of the American Control Conference, Denver, CO, USA, 1–3 July 2020; Volume 2020, pp. 1179–1184. [ Google Scholar ] [ CrossRef ]
• Li, D.; Ranjitkar, P.; Zhao, Y.; Bai, Q.; Chen, X. Improving motorway performance using variable speed limits. Proc. Inst. Civ. Eng. Transp. 2017 , 170 , 338–345. [ Google Scholar ] [ CrossRef ]
• Fondzenyuy, S.K.; Turner, B.M.; Burlacu, A.F.; Jurewicz, C. The contribution of excessive or inappropriate speeds to road traffic crashes and fatalities: A review of literature. Transp. Eng. 2024 , 17 , 100259. [ Google Scholar ] [ CrossRef ]
• Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009 , 6 , e1000097. [ Google Scholar ] [ CrossRef ]
• Madireddy, M.; De Coensel, B.; Can, A.; Degraeuwe, B.; Beusen, B.; De Vlieger, I.; Botteldooren, D. Assessment of the impact of speed limit reduction and traffic signal coordination on vehicle emissions using an integrated approach. Transp. Res. D Transp. Environ. 2011 , 16 , 504–508. [ Google Scholar ] [ CrossRef ]
• Schmaus, M.; Bawidamann, J.; Friedrich, M.; Haberl, M.; Trenkwalder, L.; Fellendorf, M.; Uhlig, J.; Lohse, R.; Pestel, E. Flüssiger Verkehr für Klimaschutz und Luftreinhaltung. 2023. Available online: https://www.umweltbundesamt.de/sites/default/files/medien/479/publikationen/texte_14-2023_fluessiger_verkehr_fuer_klimaschutz_und_luftreinhaltung.pdf (accessed on 27 November 2023).
• Casanova, J.; Fonseca, N. Environmental assessment of low speed policies for motor vehicle mobility in city centres. Glob. Nest J. 2012 , 14 , 192–201. [ Google Scholar ] [ CrossRef ]
• Rhode, J.; Wagner, P.; Ziemke, T. The Effect of Speed Limits and Traffic Signal Control on Emissions. Procedia Comput. Sci. 2022 , 201 , 568–573. [ Google Scholar ] [ CrossRef ]
• Williams, D.; Robin, N. An Evaluation of the Estimated Impacts on Vehicle Emissions of a 20mph Speed Restriction in Central London. 2013; 71p. Available online: https://www.edinburgh.gov.uk/downloads/file/25178/city-of-london-emissions-report (accessed on 30 October 2023).
• Gressai, M.; Varga, B.; Tettamanti, T.; Varga, I. Investigating the impacts of urban speed limit reduction through microscopic traffic simulation. Commun. Transp. Res. 2021 , 1 , 100018. [ Google Scholar ] [ CrossRef ]
• Tang, J.; McNabola, A.; Misstear, B.; Pilla, F.; Alam, M.S. Assessing the impact of vehicle speed limits and fleet composition on air quality near a school. Int. J. Environ. Res. Public Health 2019 , 16 , 149. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tang, J.; McNabola, A.; Misstear, B. The potential impacts of different traffic management strategies on air pollution and public health for a more sustainable city: A modelling case study from Dublin, Ireland. Sustain. Cities Soc. 2020 , 60 , 102229. [ Google Scholar ] [ CrossRef ]
• Jones, S.J.; Brunt, H. Twenty miles per hour speed limits: A sustainable solution to public health problems in Wales. J. Epidemiol. Community Health (1978) 2017 , 71 , 699–706. [ Google Scholar ] [ CrossRef ]
• Owen, B. Air quality impacts of speed-restriction zones for road traffic. Sci. Total Environ. 2005 , 340 , 13–22. [ Google Scholar ] [ CrossRef ]
• Cleland, C.L.; McComb, K.; Kee, F.; Jepson, R.; Kelly, M.P.; Milton, K.; Nightingale, G.; Kelly, P.; Baker, G.; Craig, N.; et al. Effects of 20 mph interventions on a range of public health outcomes: A meta-narrative evidence synthesis. J. Transp. Health 2020 , 17 , 100633. [ Google Scholar ] [ CrossRef ]
• Gonçalves, M.; Jiménez-Guerrero, P.; López, E.; Baldasano, J.M. Air quality models sensitivity to on-road traffic speed representation: Effects on air quality of 80 km h −1 speed limit in the Barcelona Metropolitan area. Atmos. Environ. 2008 , 42 , 8389–8402. [ Google Scholar ] [ CrossRef ]
• Bel, G.; Rosell, J. Effects of the 80 km/h and variable speed limits on air pollution in the metropolitan area of barcelona. Transp. Res. D Transp. Environ. 2013 , 23 , 90–97. [ Google Scholar ] [ CrossRef ]
• Keller, J.; Andreani-Aksoyoglu, S.; Tinguely, M.; Flemming, J.; Heldstab, J.; Keller, M.; Zbinden, R.; Prevot, A.S. The impact of reducing the maximum speed limit on motorways in Switzerland to 80 km h-1 on emissions and peak ozone. Environ. Model. Softw. 2008 , 23 , 322–332. [ Google Scholar ] [ CrossRef ]
• Cohen, S.; Christoforou, Z.; Seidowsky, R. Assessing the impact of speed limit changes on urban motorways: A simulation study in Lille, France. Transp. Res. Procedia 2014 , 3 , 915–924. [ Google Scholar ] [ CrossRef ]
• Dijkema, M.B.A.; van der Zee, S.C.; Brunekreef, B.; van Strien, R.T. Air quality effects of an urban highway speed limit reduction. Atmos. Environ. 2008 , 42 , 9098–9105. [ Google Scholar ] [ CrossRef ]
• Folgerø, I.K.; Harding, T.; Westby, B.S. Going fast or going green? Evidence from environmental speed limits in Norway. Transp. Res. D Transp. Environ. 2020 , 82 , 102261. [ Google Scholar ] [ CrossRef ]
• Lopez-Aparicio, S.; Grythe, H.; Thorne, R.J.; Vogt, M. Costs and benefits of implementing an Environmental Speed Limit in a Nordic city. Sci. Total Environ. 2020 , 720 , 137577. [ Google Scholar ] [ CrossRef ]
• Keuken, M.P.; Jonkers, S.; Wilmink, I.R.; Wesseling, J. Reduced NOx and PM10 emissions on urban motorways in The Netherlands by 80km/h speed management. Sci. Total Environ. 2010 , 408 , 2517–2526. [ Google Scholar ] [ CrossRef ]
• Zhai, Z.; Xu, J.; Song, G.; Hatzopoulou, M. Comparative analysis of drive-cycles, speed limit violations, and emissions in two cities: Toronto and Beijing. Sci. Total Environ. 2022 , 811 , 152323. [ Google Scholar ] [ CrossRef ]
• Lai, F.; Carsten, O.; Tate, F. How much benefit does Intelligent Speed Adaptation deliver?—An analysis of its potential contribution to safety and environment. Accid. Anal. Prev. 2012 , 48 , 63–72. [ Google Scholar ] [ CrossRef ]
• Archer, J.; Fotheringham, N.; Symmons, M.; Corben, B. The Impact of Lowered Speed Limits in Urban and Metropolitan Areas ; Monash University Accident Research Centre—Report 276; Monash University Accident Research Centre: Clayton, VIC, Australia, 2008. [ Google Scholar ]
• SenterNovem. Ecodriving: The Smart Driving Style ; Senter-Novem–Agency of the Ministry of Economic Affairs: Dutch, The Netherlands, 2005. [ Google Scholar ]
• Jones, S.; Musselwhite, C.; Baker, L.; Nicholass, A. Reducing speed limits in residential areas has huge public health benefits: But what do we need to do to convince the public? J. Transp. Health 2022 , 26 , 101492. [ Google Scholar ] [ CrossRef ]
• Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021 , 372 , n71. [ Google Scholar ] [ CrossRef ] [ PubMed ]

Click here to enlarge figure

Share and Cite

Fondzenyuy, S.K.; Turner, B.M.; Burlacu, A.F.; Jurewicz, C.; Usami, D.S.; Feudjio, S.L.T.; Persia, L. The Impact of Speed Limit Change on Emissions: A Systematic Review of Literature. Sustainability 2024 , 16 , 7712. https://doi.org/10.3390/su16177712

Fondzenyuy SK, Turner BM, Burlacu AF, Jurewicz C, Usami DS, Feudjio SLT, Persia L. The Impact of Speed Limit Change on Emissions: A Systematic Review of Literature. Sustainability . 2024; 16(17):7712. https://doi.org/10.3390/su16177712

Fondzenyuy, Stephen Kome, Blair Matthew Turner, Alina Florentina Burlacu, Chris Jurewicz, Davide Shingo Usami, Steffel Ludivin Tezong Feudjio, and Luca Persia. 2024. "The Impact of Speed Limit Change on Emissions: A Systematic Review of Literature" Sustainability 16, no. 17: 7712. https://doi.org/10.3390/su16177712

Article Metrics

Supplementary material.

ZIP-Document (ZIP, 82 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Open access
• Published: 04 September 2024

Targeting emotion dysregulation in depression: an intervention mapping protocol augmented by participatory action research

• Myungjoo Lee   ORCID: orcid.org/0000-0002-8301-7996 1 ,
• Han Choi   ORCID: orcid.org/0000-0003-0406-5605 2 &
• Young Tak Jo   ORCID: orcid.org/0000-0002-0561-2503 1  

BMC Psychiatry volume  24 , Article number:  595 ( 2024 ) Cite this article

Metrics details

Depression is a highly prevalent and often recurrent condition; however, treatment is not always accessible or effective in addressing abnormalities in emotional processing. Given the high prevalence of depression worldwide, identifying and mapping out effective and sustainable interventions is crucial. Emotion dysregulation in depression is not readily amenable to improvement due to the complex, time-dynamic nature of emotion; however, systematic planning frameworks for programs addressing behavioral changes can provide guidelines for the development of a rational intervention that tackles these difficulties. This study proposes an empirical and theoretical art-based emotion regulation (ER) intervention using an integrated approach that combines intervention mapping (IM) with participatory action research (PAR).

We used the IM protocol to identify strategies and develop an intervention for patients with major depressive disorder (MDD). As applied in this study, IM comprises six steps: (a) determining the need for new treatments and determinants of risk; (b) identifying changeable determinants and assigning specific intervention targets; (c) selecting strategies to improve ER across relevant theories and research disciplines; (d) creating a treatment program and refining it based on consultations with an advisory group; (e) developing the implementation plan and conducting a PAR study to pilot-test it; and (f) planning evaluation strategies and conducting a PAR study for feedback on the initial testing.

Following the steps of IM, we developed two frameworks for an art-based ER intervention: an individual and an integrative framework. The programs include four theory- and evidence-based ER strategies aimed mainly at decreasing depressive symptoms and improving ER in patients with MDD. We also developed a plan for evaluating the proposed intervention. Based on our preliminary PAR studies, the intervention was feasible and acceptable for adoption and implementation in primary care settings.

The application of IM incorporated with PAR has resulted in an intervention for improving ER in depression. While changing behavior is perceived as a challenging and elaborate task, this method can be useful in offering a clear structure for developing rational interventions. Further refinement is necessary through rigorous research.

Peer Review reports

Depression is a highly prevalent and often recurrent condition that severely impairs psychological functioning and quality of life. According to the Global Health Data Exchange, depression affects 3.8% of the world’s population and, as “a major contributor to the overall global burden of disease,” is associated with substantial societal and personal costs [ 1 , 2 ]. Due to its enormous impact on public health, the World Health Organization (WHO) predicts that depression will rank first among all causes of the burden of disease by 2030 [ 3 ]. As depression is frequently comorbid with other mental and physical disorders, it is particularly challenging to identify risk factors and develop effective interventions.

Depression is a disorder of emotion. Disordered affect is a hallmark of depressive episodes, characterized by complex but apparent abnormalities of emotional functioning [ 4 , 5 ]. Many factors may be associated with the disorder; however, its symptoms evidently indicate failures in emotional self-regulation [ 6 ]. Emotion regulation (ER) refers to an individual’s ability to modulate the intensity, frequency, and duration of emotional responses [ 7 , 8 ]. Decades of empirical research have shown that depression is associated with increases in unpleasant emotions and decreases in positive emotions [ 9 , 10 ]. It has been proposed that difficulties in ER in depression significantly contribute to dysfunctional emotions [ 10 , 11 ].

The complexity and time-dynamic nature of emotion make emotion dysregulation in depression particularly challenging to tackle. Most situations in daily life that evoke emotions are ambiguous. It remains unclear how patients can enhance their ER abilities in treatment [ 12 ]. Dysfunctional ER is a fundamental risk factor for the onset of depression and a range of psychiatric disorders [ 13 , 14 ]; however, the evidence base is diffuse and broad, as its mechanisms remain poorly specified [ 12 , 15 , 16 ]. Although some studies have developed psychological interventions to improve ER, research in this area remains limited [ 12 , 17 , 18 ]. Some have argued that teaching a wide range of ER strategies might not be effective in enhancing patients’ emotional functioning [ 12 , 17 ]. Of note, there is a lack of research on the use of art psychotherapy in this context.

An intervention mapping (IM) study systematically rooted in the evidence and theories of basic affective science is required to increase the likelihood of changing behaviors in ER. To target emotional dysregulation, a systematic, participatory, and integrated approach that benefits from efficient behavior change is crucial [ 19 ]. Accordingly, this study determines effective ways of enhancing patients’ ER capacities and developing an optimized art-based psychotherapy intervention for depression. For this purpose, we followed the standard IM protocol [ 20 ]. While developing a treatment may be time-consuming and burdensome, this study provides a straightforward, stepwise decision-making procedure. Along with its use of participatory action research (PAR), this study aims to benefit from the engagement of patients and mental health professionals in a collaborative manner. This type of collaboration is a practical and powerful tool for developing specialized interventions.

Intervention mapping protocol

This study mapped out the process of development based on IM, a program-planning framework. IM provides a step-by-step process for planning theory/evidence-based interventions from the needs to potential methods addressing those needs [ 20 , 21 ]. Since its development in the healthcare field in 1998, IM has been widely used and applications have emerged in other fields, including health promotion. It has been used to develop intervention programs to better target specific behaviors, including health, discrimination, and safety behaviors [ 22 ]. In particular, mental health researchers have largely applied the IM approach for either creating new interventions or adapting existing ones: strategies have been developed for the treatment and prevention of depression through IM, such as an internet-based intervention for postpartum depression [ 23 ], an online-coaching blended program for depression in middle-aged couples [ 24 ], a return-to-work intervention for depression [ 25 ], music therapy for depressive symptoms in young adults [ 26 ], and life-review therapy for depressive symptoms in nursing home residents [ 27 ]. The use of IM has proven to be a useful instrument for the development and optimization of treatments for depression that are tailored to different contexts and target populations.

Over the course of the development of the entire program, four perspectives characterizing IM are applied: (a) a participation approach that engages intended participants, allies, and implementers in program development, implementation, and evaluation; (b) a systems approach acknowledging that an intervention is an event occurring in a system that includes other factors influencing the intervention; (c) a multi-theory approach that stimulates the use of multiple theories; and (d) an ecological approach recognizing the relevance of social, physical, and political environmental conditions.

The IM protocol includes six core steps: (i) justifying the rationale for developing a new treatment; (ii) selecting targeted determinants and setting treatment goals; (iii) determining theoretical and empirical methods for behavior change; (iv) developing a treatment and program materials; (v) planning for adoption and implementation; and (vi) specifying the evaluation design [ 20 , 21 , 28 ]. The development process is cumulative: subsequent steps are based on completed tasks from the previous step. Figure  1 presents the six steps of IM. This article presents the details of our study methods and the results as the six steps of the IM process.

Overview of the intervention mapping (IM) process [ 20 ]

Steps 1–3 of IM: Literature review

To address Steps 1, 2, and 3, we conducted a literature review using PubMed, ProQuest, Scopus, PsycArticles, and Google Scholar. Search strategies were devised using subject headings such as “emotion regulation,” “depression,” “emotional psychopathology,” “emotion regulation therapy,” and “art psychotherapy” as appropriate for each database. Furthermore, the program planners identified and included additional free text words. Due to the heterogeneity of emotion-related processes, the search strategies for Steps 1–3 were broad [ 15 ]. Additionally, we conducted an inclusive literature review of relevant databases to identify articles related to art-based interventions for ER, limited to published articles in English. This literature study identified effective ER strategies for improving regulatory capacities in depression. We describe the theoretical details related to ER and ER strategies in the Results section.

Steps 4–6 of IM: Participatory action research combined

Steps 4–6 of IM occasionally incorporate further studies for pilot testing and refining the intervention under development. As such, our study added participatory components to the IM process. PAR is “a participatory and consensual approach towards investigating problems and developing plans to deal with them.” [ 29 ] PAR empowers research participants compared with other approaches, where study participants are often considered subjects who passively follow directions [ 30 ]. The involvement of patients, care providers, and health professionals in research design is increasingly recognized as an essential approach for improving the quality of primary care [ 31 ] and bridging the gap between research and health care [ 32 ]. Indeed, PAR has been applied in many fields and achieved successful results, particularly in the field of mental health [ 33 ].

In particular, patient involvement is a meaningful partnership with stakeholders, including patients, carers, and public stakeholders, who actively participate in improving healthcare practices [ 31 ]. Involvement can occur at different levels and commonly includes patient engagement and advisory boards [ 32 ]. We conducted participatory action studies to combine systematic studies with the development of practical treatments [ 33 ] and anticipated the benefits of experiential knowledge. Figure  2 elaborates on how we incorporated PAR in the IM framework. It also presents our strategies to address the IM protocol and the results from each step. As described in Fig.  2 , the PAR in this current study comprises three phases:(a) consultation with an advisory board; (b) initial testing of intervention; and (c) mixed methods feedback studies using focus group interviews and survey research.

Study procedure combined with PAR, strategies applied for each step, and results for each step

Noted. The figure specifies strategies to adopt in addressing the six steps of IM protocol and the actions for each step. It represents how IM can be applied and how it can augment its protocols through PAR. In the application of IM, this study relied on literature research and empirical studies: we conducted a literature study to address Steps 1–3 and combined the participatory action approach with IM methodology to address Steps 4–6

(a) PAR 1: Consultation with the advisory board

First, we established an advisory board that included a psychiatrist, an expert on methodology, a trained integrative medicine professional, and a professor in a graduate art psychotherapy program. The advisory board provided feedback at the individual level and comments during subsequent consultations. We engaged and managed the advisory board throughout Step 4, the intervention development process.

(b) PAR 2: Initial testing of intervention being developed 

In addition, we conducted a participatory action study to facilitate patient engagement and elicit their voices in a collaborative relationship with researchers. Based on voluntary participation, this study aimed to pretest art-based ER strategies and treatment designs. We conducted an art therapy program as part of routine inpatient therapeutic programs involving willing patients. The participants’ reports of their experiences during the sessions were obtained using structured questionnaires and unstructured interviews. For research purposes, we conducted a retrospective chart review for therapeutic sessions between February 2023 and February 2024. This review was approved by the Institutional Review Board of Kangdong Sacred Heart Hospital (IRB no. 2024–02-019) and exempted from requiring patients’ informed consent because it was part of a routine clinical practice.

(c) PAR 3: A mixed-method approach

In this study, we employed a mixed-methods approach to plan evaluation strategies by combining a quantitative online survey with focus group interviews. The primary aim of this study is to ensure that the intervention developed in Step 4 can be adopted and maintained over time. For this purpose, we are gathering feedback regarding the initial interventions from clinic staff, consisting of nurses and psychiatrists. This PAR study is currently ongoing and will last for four months. At the environmental level of the organization, the process will be managed to best leverage the intervention in primary care settings. This study was approved by the Institutional Review Board of Kangdong Sacred Heart Hospital (IRB no. 2023–12-002). PAR 2 and PAR 3 are currently being conducted; the results of those studies will be available after their completion.

This section focuses on the explanation of outputs obtained through the IM protocol. The details of the theoretical and empirical bases, designed frameworks, and strategies for the implementation and evaluation of the program are categorized into six steps:

Step 1. Needs and Logic for the Program

For the first step, we identified the target group and analyzed their determinants. This step included determining the rationale and need for a new art-based ER intervention for depression. The target population comprised patients diagnosed with major depressive disorder (MDD). Predefined behaviors targeted were core symptoms of major depression, namely, consistent depressed mood and anhedonia [ 6 ].

Theoretical evidence

Prior research has highlighted difficulties in ER contributing to the etiology and maintenance of numerous psychiatric symptoms, such as depression, chronic anxiety, post-traumatic stress disorder, eating disorders, and worry [ 15 , 34 , 35 , 36 , 37 , 38 , 39 , 40 ]. In particular, research on depression has emphasized that apparent failure to modulate emotions is a hallmark of this disorder [ 6 ] and has attempted to link it to emotional abnormalities in depression [ 10 , 11 ]. ER, which influences the onset, magnitude, and duration of emotional response [ 41 ], is a distinct and differentiated higher-order construct from emotion itself (i.e., fear, anxiety, and depression) at different levels of analysis (e.g., behavioral or neural) [ 42 , 43 ]. From this perspective, ER is an important determinant affecting lower-order factor variability, whereas emotion determines variance downwards in the lower-order indicators [ 42 ].

A literature review revealed that ER difficulties play a role in understanding psychological health in major depression. This suggests the importance of altering problematic patterns of emotional reactivity in depression and identifies emotion dysregulation as a determinant of the predefined target behaviors [ 17 , 44 , 45 , 46 , 47 ]. According to imaging studies utilizing functional magnetic resonance imaging (fMRI), functional abnormalities in specific neural systems support the processing of emotion and ER in patients with depressive disorders [ 6 ]. Moreover, decades of empirical evidence supports the notion that depressive symptoms, characterized by consistently elevated depressed mood and relatively low positive mood, are associated with difficulties in ER [ 9 , 10 , 16 ]. Our review allowed us to analyze and specify the determinants of depressive symptoms (Fig.  3 ). Without this analysis, it would be challenging for psychological treatments to address emotion dysregulation in MDD.

Summary of the determinants influencing symptoms of major depression

Needs assessment for a new intervention

Although emotion dysregulation is a critical target in psychological treatments, intervention research examining ER is limited [ 18 , 48 ]. Psychotherapeutic approaches, including cognitive-behavioral and acceptance-based behavioral treatments, have positive effects on overall ER, and studies suggest that these improvements may mediate further improvements for psychiatric outcomes [ 18 , 48 ]: examples include cognitive behavioral therapy approaches (CBT) [ 49 , 50 ], acceptance and commitment therapy (ACT) [ 51 ], dialectical behavioral therapy (DBT) [ 52 , 53 ], and acceptance-based behavioral therapy (ABBT) [ 54 ]. However, most research assessing treatment efficacy precludes making any decisions about clinical mechanisms essential for improving ER. This is because they examine the impact of non-ER-focused interventions or interventions to target ER as part of a comprehensive program [ 18 , 48 ]. Due to the multi-component nature of the interventions, the specific components contributing to changes in ER remains unclear and whether the changes underlie improvements in other distressing symptoms has not yet been clarified. Thus, efforts to identify and inform the development of interventions leading to adaptive ER based on these studies are limited.

At present, patients who have distress disorders, such as generalized anxiety disorder (GAD), MDD, and particularly GAD diagnosed along with comorbid depression, often fail to respond well or experience sufficient gains from treatments: however, the reason for their lack of response is unknown [ 17 , 55 ]. Between 50 and 80% of patients receiving interventions for emotional disorders achieve the status of “responder.” [ 17 ] Between 50 and 60% of GAD patients showed meaningful improvement in response to treatment with traditional CBT [ 55 ]. While ER-focused interventions, such as the Unified Protocol (UP) [ 56 ], Emotion Regulation Group Therapy (ERGT) [ 57 ], and enhanced CBT emphasizing ER [ 58 ] were found to be effective in improving ER, research investigating these remains limited [ 18 , 59 , 60 ]. No substantial changes were found in the essential dimensions of ER after the application of several ER-focused interventions, implying that these were not present in a sufficient dose to promote ER [ 53 , 61 , 62 ]. Further, recent research identifying treatment response predictors for ERGT showed relatively few significant predictors [ 63 ]. In particular, the findings from a study that examined a treatment designed to enhance inpatient CBT for depression suggest that the addition of ER skills to CBT may not sufficiently change ER, although improvements were noted in ER strategies and depressive symptoms [ 58 ]. Another problem arises from the manualized CBT protocols, which are distinct and complex to use [ 17 , 64 ]. These protocols make it difficult to access and use CBT.

The limitations of the current interventions suggest the need for developing an ER-specific treatment. Designing more effective and targeted interventions requires a specific understanding of affective science to provide a broad framework for ER treatments. For example, recently, it has been identified that emotions can be generated and regulated not only through a top-down process but also through a bottom-up process: [ 65 ] current models of emotion generation and its regulation are based on these two processes, which are opposed but interactive [ 66 ]. The top-down mechanism is based on a view that focuses on cognition, where either individuals’ goal states or cognitive evaluations are thought to influence the variations in their emotional responses [ 67 ]. These processes are mapped to prefrontal cortical areas. Meanwhile, bottom-up mechanisms refer to processes based on a stimuli-focused view: in this mode of processing, emotions are mostly elicited by perceptions [ 68 ]. In everyday life, emotion can be processed through interactions between the bottom-up and top-down mechanisms [ 69 ].

Most research to date, however, has focused on top-down ER strategies, and few studies have focused on bottom-up regulation procedures [ 65 ]. In particular, CBT-based treatments, which are mainstream psychotherapies, focus on instruction in an array of cognitive means of coping with emotions; CBT traditionally tends to deal more directly with cognitive rather than emotional processes. One top-down strategy is cognitive reappraisal, an active component of most CBT-based treatments [ 70 ]. However, studies suggest that relying primarily on this strategy may be less effective for certain disorders, including depression, than treatments employing a flexible approach [ 65 ]. Such an approach would be straightforward and essential for researchers as they synthesize different research results, such as findings concerning bottom-up ER and its clinical implications for the investigation of interventions.

One intervention approach to bottom-up experiential ER is art psychotherapy. This type of treatment, which targets emotion dysregulation, may hold promise for improving ER in cases of depression. Patients with depression can benefit from experiential ER that emphasizes bottom-up means of coping with their emotional experiences over the course of art-based ER intervention. This perspective is supported by behavioral and neurocognitive findings indicating difficulties in top-down regulatory processes in individuals with depression [ 71 , 72 , 73 , 74 ]. Research examining neural activities between individuals with and without depression indicated different patterns between them: when downregulating negative emotions, individuals with depression show bilateral prefrontal cortex (PFC) activation, whereas individuals without depression show left-lateralized activation [ 74 ]. When given an effortful reappraisal task, moreover, the relationship patterns of individuals with depression between activation in the left ventrolateral PFC and the amygdala are different from those of individuals without depression. These findings indicate that the pathophysiology of depression underlies struggles of downregulation [ 74 ].

Thus, it is vital to design a new intervention for depression that focuses not only on top-down ER but also on bottom-up ER. In particular, this study examines art-based ER in the form of a client-centered and experiential psychotherapeutic approach allowing patients to attempt top-down and bottom-up regulation. While pursuing active engagement in art-based ER practices, patients can process their emotional experiences in a way that produces greater fine-tuning and depth. Art-based treatment is open and non-interventional as well as less demanding cognitively, enabling it to reach a diverse population with depressive symptoms. More promisingly, art-based ER primarily deals with visible and tangible works leading to visual representations. Emotional memory is perceptual [ 75 ], implying that art-based practices can influence its retrieval and manipulative process: the artworks that patients make in treatments are visual representations that are identical or similar to their emotional experiences. Importantly, creation involves colors, images, and spaces acting as new stimuli, allowing patients to manipulate and generate new emotions through a bottom-up process. As processes of emotion generation interact with those of ER [ 67 ], an art-based experiential approach can facilitate adaptive ER, potentially benefiting individuals who have emotional dysfunction.

However, few studies have explored ER in depression within the field of art psychotherapy [ 76 ]. The therapeutic strategies applied in relevant studies [ 77 , 78 , 79 ] are not explicitly identified or targeted with respect to the mechanisms of ER. For instance, earlier literature tested the effects of art therapy on ER in psychiatric disorders; most of these approaches focused on improving psychopathological symptoms related to specific disorders and considered ER to be a secondary therapeutic outcome. Thus, we identified a need to develop an effective art-based intervention specifically targeting emotion dysregulation in major depression.

Step 2: Formulation of change objectives

The second step required the specification of intervention goals, which involved moving from understanding what influences depressive symptoms, especially in terms of emotional abnormalities in depression, to clarifying what needs to be changed. Based on the needs assessment, the overall expected outcome was “a decrease in depressive symptoms and an improvement in ER.” In this process, the analysis of the determinants in Step 1 resulted in selecting key determinants to target, which were provided by a comprehensive review of the empirical literature and research evidence. It is difficult to understand generative and regulatory emotion processes that are enacted internally without the instigation of extrinsic stimuli [ 80 ]. Thus, it can be challenging to identify the right determinants to target and design an effective treatment that addresses problems related to ER. Based on our review, we determined and chose four important and changeable determinants and further divided them into five key determinants (see Table  1 ).

To apply IM, the construction of matrices of change consisting of performance and change objectives forms the basis for program development [ 20 , 81 ]. Overall, the program objectives were subdivided into performance objectives expected to be accomplished by the target group in the proposed intervention. While drawing on the key determinants and performance objectives, more general objectives, namely, change objectives, were formulated. The result of Step 2 is this change matrix, which further forms the basic factors for designing the intervention for major depression.

Step 3: Theory- and evidence-based strategies selection

In IM, Step 3 entails selecting theoretically grounded and evidence-based methods and strategies. For this process, we first conducted a comprehensive review of theories and empirical studies for therapeutic strategies, including the following characteristics: (i) they need to be confirmed as an efficient ER strategy based on empirical research evidence; (ii) they need to be effective not only in decreasing depressive symptoms but also in improving ER capacities of patients; and (iii) they can be translated into art-based practices. In iterations of reviewing theories related to and research evidence with regard to emotion regulatory strategies, we identified appropriate, theoretically sound therapeutic strategies for at least one program target.

Once an ER method was selected, we translated this method into art-based emotion regulation (ABER) strategies for practical applications. Practical applications refer to the practical translation of the chosen behavior change methods [ 19 , 20 , 21 , 81 ]. The end product of Step 3 is an initial set of theory- and evidence-based strategies selected and translated to address emotion dysregulation in major depression. Table 2  lists the strategies with supporting evidence and applications: art-based distraction, art-based positive rumination, art-based self-distancing (SD), and art-based acceptance. Based on an integrative view of emotional processing, which posits interactions between top-down and bottom-up systems [ 67 , 69 , 82 , 83 ], these strategies aim to modulate emotions through the use of top-down and bottom-up mechanisms.

In particular, as art-based ER involves visual-spatial processing that could exert influence as new sets of stimuli, this approach could lead to a more experiential bottom-up ER. For instance, distraction and cognitive defusion are usually considered cognitive forms of ER; however, both are translated and applied to art-based strategies. Individuals’ performance in art-based ER would differ from that on a given cognitive task, as their immersion experiences in the artistic and creative process involve the generation of colors, images, and spatial features, which may elicit new bottom-up processing. This may be associated with the superior ER effects of art-based distraction, as shown in some studies that compared the ER effects of artistic activities with those of non-artistic activities, such as completing verbal puzzles [ 98 , 99 , 100 ].

In addition, art-based SD promotes intuitive and experiential ER. Individuals are trained to adopt a self-distanced perspective in some treatments while reflecting on their emotions, such as mindfulness-based stress reduction (MBSR) and ERT. They meditate to take a decentered stance. Art-based SD may help those who have difficulty creating an internal distance. As individuals create visual forms of their inner feelings and thoughts, a spatially generated distance from the artworks representing their experiences allows them to adopt and maintain a more self-distanced perspective. As such, art-based SD is more intuitive but requires less mental energy. Importantly, this art-based experiential distancing may reconstrue individuals’ appraisals by facilitating a bottom-up mechanism.

Step 4: Program development

Step 4 concerns creating an actual program plan, leading to the ABER intervention model proposed in the current study. The intervention's elementary components, organization, and structure were created based on the findings of the preparation steps (Steps 1–3). Once the list of therapeutic strategies and their practical applications was generated, we designed a structured intervention framework that would be feasible and realistic to deliver in primary care settings.

The intervention framework developed in Step 4 is based on the process model of ER [ 7 ], supported by considerable empirical research [ 101 , 102 , 103 ]. Based on the extended model, a series of steps involved in the process of regulation with different ER strategies are considered while designing the conceptual framework. Accordingly, the primary areas of the intervention involve emotion perception, attention, and cognition. We developed specific art-based ER strategies, focusing primarily on antecedent- rather than response-focused regulation. Further, this intervention is meant to complement the process model in a framework that is designed to apply one or more strategies in a single session: this would be ideal for improving ER in real life, as current research on ER has found that people generally try multiple strategies simultaneously [ 104 ], whereas the process model examines a within-situation context, within which a single ER strategy is utilized [ 12 ]. In addition, we find that this treatment will be effective in improving ER as it attempts both top-down and bottom-up ER: actively engaging in artworks through the use of the body, a patient can apply experiential self-focus [ 64 ]. In treatment with art-making, patients can be provided with sufficient time and space to find personal meaning in their experiences and process emotions, which enables them to achieve change.

Table 3 presents an overview of the proposed intervention frameworks. As shown, we designed two frameworks to guide the intervention: an individual framework for short-term intervention and an integrative framework for long-term intervention. Each style of the ABER model draws on a different implementation design to build the framework, and each model has slightly different aims. In Step 4, the advisory board reviewed the draft frameworks, including the determinants, performance and change objectives, and therapeutic strategies. The advisory board acted as a support group throughout the review process, helping tailor the program to the target population. In response to the board’s reviews, supplementary resources were added.

Individual framework

First, a plan for an individual framework was devised that accounted for the scope and phase of a short-term intervention. As shown in Fig.  4 , this framework focuses on producing initial or short-term behavioral changes pertaining to achieving short-term clinical efficacy. That is, the individual model does not aim only at emotional changes in patients, such as increases or decreases in specific emotions. The therapeutic aim is not set in an emotion-specific manner, but in terms of effectiveness, it relates to the use of regulatory strategies [ 105 ]. Accordingly, an expected outcome is to increase the quantity and frequency of adaptive ER strategies. Patients are trained in rudimentary ER skills, including one or several combination ABER strategies, as designed in the previous step. These practices aim to enhance attentional, followed by cognitive control. The expected duration of individual sessions is around 1–1.5 h.

Individual intervention model diagram. Noted. The panel shows the individual intervention model in an inpatient setting as an example: each patient (patient i ) has a different time of admission (t 0 ) and inpatient discharge (t d ). Thus, the number of participating patients can differ per session. During the hospital stay, patients are trained in rudimentary emotion regulation (ER) skills, including one or a combination of several art-based ER strategies (aber i ). The application of the therapeutic strategies is flexible: it depends on the patient’s cognitive functions, depressive symptoms, and severity of the symptoms. The time of inpatient discharge (t d ) affects each patient’s treatment duration

Integrative framework

While an individual framework comprises a single phase, an integrative framework includes stepwise sequential phases. In addition to skill development in the individual treatment, three phases of the integrative model are designed to foster adaptive motivational responses and cognitive-behavioral flexibility, which enables patients to achieve greater emotional clarity [ 106 ]. In the integrative treatment, all three phases are performed for 6–12 weeks.

The first phase of the integrative model begins with psychoeducation, in which the patient is taught the concept of ER and the importance of identifying his or her habitual reactions, such as in terms of rumination and dampening [ 91 ], that have characterized his or her life. This therapeutic process is important because ER is an automatic process requiring the consideration of motivation [ 107 ]. Psychoeducation regarding ER and monitoring patients’ responses to emotional experiences precede the skill development procedure. For instance, for patients’ self-monitoring, retrospective self-report questionnaires can capture data on ER skill use. While these methodologies are easy to use and cost-efficient [ 108 ], they are demanding tools for use in capturing natural fluctuating patterns in ER [ 109 ]. As an alternative, ecological momentary assessment can be used in treatment to capture situational context and adaptiveness of the skill use [ 108 ]. In addition to patients’ self-monitoring, a psychotherapist should monitor their emotional responses during and between therapy sessions: psychotherapists function as human raters. Because self-monitoring may not be feasible for all patients, assessing the typical patterns with which patients use maladaptive emotion regulatory strategies is important. Specifically, therapists need to assess a patient’s ER repertoire: the quantity of ER strategies, the frequency of strategy use, and how the patient’s strategy use changes.

The second phase entails adopting and implementing ER strategies with processes resembling those of the individual model. These processes entail the selection and repetition of adaptive strategies. They differ from the individual model in that the duration of Phase II can vary from one patient to another depending on the severity of depressive symptoms and the frequency of maladaptive strategies used. The ER practices delivered in Phase II are art-based tasks through which therapists and patients explore and try adaptive strategies. As shown in Fig.  5 , the intervention program includes four ABER strategies selected and translated in Step 3: art-based distraction, art-based SD, art-based positive rumination, and art-based acceptance. The patients work with therapists in 4–8 1.5-h sessions to engage in art-based practices.

Summary steps and components for the integrative intervention model

Finally, the integrative framework includes a third phase for evaluation. While the previous sessions in Phase II focus on skill development, the sessions in Phase III focus on assessing changes in patients. All individual progress in ER is tracked and monitored. In this task, therapists help patients assess changes in their emotion-regulatory skill use and their achievements in terms of self-perception, effectiveness, and adaptiveness. Patients are given opportunities to take a broad view of the changes in their artworks during all treatment phases. Furthermore, patients receive a few tasks as homework to briefly review their strategy use in daily life from the beginning of the treatment until the current moment. The review process helps them assess their progress and supports their strengthening. It takes 6–12 weeks to complete the integrative treatment course, depending on the clinical impression. For instance, the duration of Phase II is expected to take 4–8 weeks, according to the clinical impression. A therapist or clinician renders his or her impression regarding the degree of the patients’ severity of depressive symptoms, use of maladaptive ER strategies, willingness to participate in the intervention, and insight into their treatment.

Step 5: Adoption and implementation

Implementation is an essential aspect of program development. In Step 5 of IM, the focus is on planning the adoption and implementation of the proposed intervention. This process is required at the environmental level [ 21 ] and ensures successful adoption and sustainable use in collaborating organizations. Thus, pilot tests can be conducted to gain practical insights into implementation decisions and refine the intervention. Using a PAR framework, we pilot-tested the individual model to ensure that the intervention is appropriate and helpful for patients. This PAR pilot study was performed to inform future practices while connecting intervention research with actual action in a primary care setting.

The advisory group’s results, which indicated that the intervention needed to be sufficiently pliable to be used in a variety of primary care settings, informed and supported the step for pretesting. Implementation was prepared in a primary care setting, in which the program was pretested with a steering group of psychiatrists, nurses, and an art psychotherapist. Two clinicians were in charge of informing the intervention program and facilitating patient involvement. The therapist, who had received appropriate training and instruction, was responsible for delivering the intervention and supporting all practical aspects of patient engagement. With support from the therapist, the patients were in charge of applying one or a combination of two strategies in therapeutic sessions.

We performed this initial testing in a psychiatric ward in Seoul. Between February 2023 and February 2024, during the first two phases of the pilot testing, approximately 24 sessions were conducted, and 45 inpatients, including 16 patients with depressive disorders, voluntarily participated in the program. At the end of each session, the participants were asked to report their experiences through free narratives and complete a short questionnaire survey (quantitative and free-text comments) that provided additional information regarding their involvement. The mean time expenditure for the patients was 1.1 h (SD: 18.0; range: 0.5–2). Patients’ emotional experiences were reflected in their artworks, and Fig.  6 shows a short overview of their art products. The detailed findings from these pilot trials are outside the scope of the IM protocol and will be available in a future publication.

Examples of the art products of the participating patients with depression. Noted. Figure 6 briefly outlines patient engagement through their artworks made during the treatment sessions in the first pilot phase: a shows an artwork a patient made in a treatment session, which applied art-based acceptance; b shows an artwork showing a patient’s reflection on his experience, which applied art-based self-distancing and acceptance; c and d show artworks in which patients apply art-based positive rumination and distraction. Different art materials were provided in each session depending on the ER strategies used. The art-based practices of ER promoted relaxation and expression of the patient’s inner feelings and thoughts

Step 6. Evaluation plan

The sixth step of IM is the planning of evaluation strategies to assess the potential impacts of the proposed intervention [ 20 ]. For this purpose, we designed two phases based on a PAR framework: patient feedback and expert feedback. The rationale for this plan was that comprehensive evaluations could investigate the necessity of refinement and what is needed to produce a more feasible and effective intervention. In particular, we expected that the engagement of patients as well as health professionals in the evaluation process would integrate the organizational perspective into patient-oriented quality improvements. From these two phases, we developed questions and measures for evaluation, conducting preliminary PAR studies to determine the feasibility and efficacy of the complete program. Table 4 presents the evaluation strategies for gaining patient and expert feedback. Meanwhile, Table  5 presents an overview and timeline of PAR 2 and PAR 3.

First, we developed a set of patient-reported outcome measures to obtain patient feedback. Quantitative assessments of treatment satisfaction, perceived helpfulness of treatment, and perceived difficulty were conducted following the end of a therapeutic session. Patient evaluations must be carried out regularly during treatment to assess the efficacy of the integrative model. At the end of the program, unstructured or semi-structured interviews are recommended to explore patients’ experiences of the treatment process. In addition, we planned a two-phase mixed-methods study to obtain feedback from participating healthcare professionals using an online survey and focus group interviews. The assessments included process measures, such as perceived difficulty, program appropriateness, and recommendations for improvements to its implementation on a professional level. A web-based survey was disseminated among clinicians and nurses to assess the feasibility of the intervention. Together, this enabled us to increase the time efficiency and cost-effectiveness of the evaluation process.

Feasibility was assessed in five ways. First, the feasibility with which patients participated in the program was described. In our preliminary study, for instance, we calculated the percentage of patients approached for program participation relative to those who did not. Second, the feasibility of retaining patients in a treatment session was reported. To capture the feasibility of retention in treatment, we calculated the percentage of patients who failed to complete treatment compared with the percentage of those who completed it. Third, the feasibility of administering treatment was measured with a self-reported survey of patients’ perceived difficulty in participation and a survey of healthcare professionals’ perceived difficulty in implementation. To report the feasibility of administering treatment, we calculated the mean hours a patient spent in completing treatment. In addition to feasibility, acceptability was operationalized in three ways: a quantitative self-report survey of patient satisfaction, patient perceptions of helpfulness of treatment, and patient willingness to recommend program participation were used. In our preliminary study, we developed responses for the patient survey and calculated the means and standard deviations for each item.

We received patient feedback in the first two pilot phases (PAR 2), and the results showed that the intervention program was feasible and acceptable for implementation in the primary care setting (the mean scores were as follows: Treatment satisfaction = 4.82, Perceived helpfulness of treatment = 4.57, Perceived difficulty = 4.45). The patients provided further recommendations for improved intervention in free-text comments. In addition to this patient feedback, we began conducting PAR3 in February 2024. The feedback research is being conducted through an online questionnaire that includes multiple-choice questions and open-ended questions, with focus group interviews being conducted virtually through Zoom. The results for PAR 2 and PAR 3 will be reported in separate articles.

In this paper, we proposed conceptual frameworks for an intervention that targets emotion dysregulation in depression. IM was used as the conceptual protocol to develop the intervention. To the best of our knowledge, this is the first art-based ER intervention incorporating previous theories, research evidence, and review data in relation to affective science and intervention research, combining PAR components with IM. We developed the intervention following the rationale and stepwise process of IM, which identifies theory- and evidence-based strategies to address key barriers to ER. In addition, to evaluate the developed intervention, preliminary PAR studies were conducted, including the acceptability of the trials and the ABER intervention to patients; the rate of recruitment, attendance, and attrition; perceived difficulties in intervention implementation; and psychological outcomes. Consequently, the intervention is theoretically underpinned and supported by empirical evidence regarding ER and the results of our pilot studies.

The current study benefits from integrating the PAR approach into the IM framework in two ways. First, using PAR studies in the IM resulted in the cogeneration of knowledge among academic researchers, implementers, and the intended participants. PAR ensured experiential knowledge to deliver content that addressed difficulties in ER in collaborative partnerships. Another contribution was enhancing the feasibility and acceptability of the proposed intervention. In particular, preliminary PAR studies helped investigate whether modifications were needed before the intervention’s adoption. Even though IM is a time-consuming process, the use of PAR made it more cost-effective and time-efficient.

In addition to these strengths, it is crucial to acknowledge and affirm the study’s limitations. First, the current study offers only preliminary evidence for the given conceptual framework. Although the proposed intervention may precisely target emotional dysfunction in depression, such as in the restrictive use of adaptive ER skills with repetitive use of maladaptive strategies, the integrative and individual frameworks of ABER have not been evaluated through randomized clinical trials. As the current study pilot-tested the intervention in an inpatient setting that served an acute, transdiagnostic population, implementers could extend the use of these frameworks by performing a fine-grained analysis of treatment contexts (e.g., by adapting the model for depressed outpatients in primary care). As such, the intervention must be examined and refined on the basis of the results of empirical studies on multidisciplinary design. In addition, this article did not examine the therapists’ capability of delivering treatment, fidelity of implementation, and feasibility of measuring tools. Intervention researchers interested in these variables are encouraged to extend our models by testing the broad contextual variables that influence its process. Similarly, further research is required to investigate standardized forms of assessment in treatment (e.g., a measurable rating scale for patient monitoring) to increase the efficiency of the intervention.

Conclusions

This article proposes empirical and theoretical intervention frameworks that can improve ER in depression. This IM study is unique, as the development process incorporates PAR components. Moreover, the intervention consists of four art-based regulatory strategies that enrich the present literature on intervention research targeting dysfunctional ER in major depression. Our participatory action studies demonstrate that, in a primary care setting, the individual protocol is feasible and acceptable for implementation. This result represents a potential step forward toward filling a gap in current mental health treatments for patients with MDD. Despite the tiresome and time-consuming process of intervention development, the application of IM augmented by PAR is helpful in optimizing chances for an effective behavior change. Further testing is required to assess the impact of the therapeutic program proposed in this study.

Availability of data and materials

The author confirms that the data generated or analysed during this study are included in this published article: however, raw datasets are not publicly available due to local legal restrictions. Since the data being generated by PAR2 and PAR3 are outside the scope of the current intervention mapping study, they are available elsewhere.

Abbreviations

Art-based emotion regulation

Cognitive-behavioral therapy

• Emotion regulation

Emotion Regulation Group Therapy

Generalized anxiety disorder

• Intervention mapping

Mindfulness-based stress reduction

Major depressive disorder

• Participatory action research

The Self-Assessment Manikin

Self-distancing

World Health Organization

Collaborators GBDMD. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: a systematic analysis for the global burden of Disease Study 2019. Lancet Psychiatry. 2022;9(2):137–50.

Article   Google Scholar  

World Health Organization. Depressive disorder(depression) . http://www.who.int/news-room/fact-sheets/detail/depression . Accessed 4th March 2024.

Malhi GS, Mann JJ. Depression Lancet. 2018;392(10161):2299–312.

Article   PubMed   Google Scholar  

Rottenberg J. Emotions in depression: What do we really know? Annu Rev Clin Psychol. 2017May;8(13):241–63.

Thompson RJ, Boden MT, Gotlib IH. Emotional variability and clarity in depression and social anxiety. Cogn Emot. 2017Jan 2;31(1):98–108.

American Psychiatric Association. Diagnostic and statistical manual of mental disorders (5th ed.). 2013. https://doi.org/10.1176/appi.books.9780890425596 .

Gross JJ. The emerging field of emotion regulation: An integrative review. Rev Gen Psychol. 1998Sep;2(3):271–99.

Gross JJ. Emotion regulation: Conceptual and empirical foundations. Handbook of emotion regulation. 2014;2:3–20.

Google Scholar  

Khazanov GK, Ruscio AM. Is low positive emotionality a specific risk factor for depression? A meta-analysis of longitudinal studies. Psychol Bull. 2016Sep;142(9):991.

Article   PubMed   PubMed Central   Google Scholar  

Vanderlind WM, Everaert J, Joormann J. Positive emotion in daily life: Emotion regulation and depression. Emotion. 2022Oct;22(7):1614.

Liu DY, Thompson RJ. Selection and implementation of emotion regulation strategies in major depressive disorder: An integrative review. Clin Psychol Rev. 2017Nov;1(57):183–94.

Southward MW, Sauer-Zavala S, Cheavens JS. Specifying the mechanisms and targets of emotion regulation: A translational framework from affective science to psychological treatment. Clin Psychol Sci Pract. 2021Jun;28(2):168.

Berenbaum H, Raghavan C, Le HN, Vernon LL, Gomez JJ. A taxonomy of emotional disturbances. Clin Psychol Sci Pract. 2003;10(2):206.

Jazaieri H, Urry HL, Gross JJ. Affective disturbance and psychopathology: An emotion regulation perspective. J Exp Psychopathol. 2013Dec;4(5):584–99.

Gross JJ, Jazaieri H. Emotion, emotion regulation, and psychopathology: An affective science perspective. Clinical psychological science. 2014Jul;2(4):387–401.

Joormann J, Stanton CH. Examining emotion regulation in depression: A review and future directions. Behav Res Ther. 2016Nov;1(86):35–49.

Barlow DH, Allen LB, Choate ML. Toward a unified treatment for emotional disorders. In The Neurotic Paradox, Volume 1. New York: Routledge; 2020. p. 141–166.

Gratz KL, Weiss NH, Tull MT. Examining emotion regulation as an outcome, mechanism, or target of psychological treatments. Curr Opin Psychol. 2015Jun;1(3):85–90.

Stutterheim SE, van der Kooij YL, Crutzen R, Ruiter RAC, Bos AER, Kok G. (2023). Intervention mapping as a guide to developing, implementing, and evaluating stigma reduction interventions. Stigma and Health. Advance online publication. https://doi.org/10.1037/sah0000445 .

Eldredge LK, Markham CM, Ruiter RA, Fernández ME, Kok G, Parcel GS. Planning health promotion programs: an intervention mapping approach. New York: Wiley; 2016.

Kok G, Bartholomew LK, Parcel GS, Gottlieb NH, Fernández ME. Finding theory-and evidence-based alternatives to fear appeals: Intervention Mapping. Int J Psychol. 2014Apr;49(2):98–107.

Kok G, Peters LW, Ruiter RA. Planning theory-and evidence-based behavior change interventions: a conceptual review of the intervention mapping protocol. Psicologia: Reflexão e Crítica. 2018;30.  https://www.scielo.br/j/prc/a/TkCBLGGQb7JRQRBbrpnjxgf/?lang=en&format=html .

Drozd F, Haga SM, Brendryen H, Slinning K. An internet-based intervention (Mamma Mia) for postpartum depression: mapping the development from theory to practice. JMIR research protocols. 2015;4(4): e4858.

Kim SS, Gil M, Kim D. Development of an online-coaching blended couple-oriented intervention for preventing depression in middle adulthood: An intervention mapping study. Front Public Health. 2022;10: 882576.

Wisenthal A, Krupa T. Using intervention mapping to deconstruct cognitive work hardening: a return-to-work intervention for people with depression. BMC Health Serv Res. 2014;14:1–11.

Aalbers S, Vink A, Freeman RE, Pattiselanno K, Spreen M, van Hooren S. Development of an improvisational music therapy intervention for young adults with depressive symptoms: An intervention mapping study. Arts Psychother. 2019;65: 101584.

van Venrooij I, Spijker J, Westerhof GJ, Leontjevas R, Gerritsen DL. Applying intervention mapping to improve the applicability of precious memories, an intervention for depressive symptoms in nursing home residents. Int J Environ Res Public Health. 2019;16(24):5163.

Roozen S, Stutterheim SE, Bos AE, Kok G, Curfs LM. Understanding the social stigma of fetal alcohol spectrum disorders: from theory to interventions. Found Sci. 2020May;29:1–9.

Bowling A. Research methods in health: investigating health and health services. UK: McGraw-hill education; 2014.

White MA, Verhoef MJ. Toward a patient-centered approach: incorporating principles of participatory action research into clinical studies. Integr Cancer Ther. 2005Mar;4(1):21–4.

Haesebaert J, Samson I, Lee-Gosselin H, Guay-Bélanger S, Proteau JF, Drouin G, Guimont C, Vigneault L, Poirier A, Sanon PN, Roch G. “They heard our voice!” patient engagement councils in community-based primary care practices: a participatory action research pilot study. Research involvement and engagement. 2020Dec;6:1–4.

Koskinas E, Gilfoyle M, Salsberg J. Exploring how patients, carers and members of the public are recruited to advisory boards, groups and panels as partners in public and patient involved health research: a scoping review protocol. BMJ Open. 2022Apr 1;12(4): e059048.

Dold CJ, Chapman RA. Hearing a voice: Results of a participatory action research study. J Child Fam Stud. 2012Jun;21:512–9.

Aldao A, Gee DG, De Los Reyes A, Seager I. Emotion regulation as a transdiagnostic factor in the development of internalizing and externalizing psychopathology: Current and future directions. Dev Psychopathol. 2016;28(4pt1):927–46.

Garnefski N, Kraaij V. Relationships between cognitive emotion regulation strategies and depressive symptoms: A comparative study of five specific samples. Personality Individ Differ. 2006Jun 1;40(8):1659–69.

Gross JJ, Muñoz RF. Emotion regulation and mental health. Clin Psychol Sci Pract. 1995;2(2):151.

Mennin DS, Holaway RM, Fresco DM, Moore MT, Heimberg RG. Delineating components of emotion and its dysregulation in anxiety and mood psychopathology. Behav Ther. 2007Sep 1;38(3):284–302.

Berking M, Wupperman P. Emotion regulation and mental health: recent findings, current challenges, and future directions. Curr Opin Psychiatry. 2012Mar 1;25(2):128–34.

Cloitre M, Khan C, Mackintosh MA, Garvert DW, Henn-Haase CM, Falvey EC, Saito J. Emotion regulation mediates the relationship between ACES and physical and mental health. Psychol Trauma Theory Res Pract Policy. 2019Jan;11(1):82.

Putnam KM, Silk KR. Emotion dysregulation and the development of borderline personality disorder. Dev Psychopathol. 2005Dec;17(4):899–925.

Gross JJ, editor. Handbook of emotion regulation. Hoboken: Guilford publications; 2013.

Cisler JM, Olatunji BO, Feldner MT, Forsyth JP. Emotion regulation and the anxiety disorders: An integrative review. J Psychopathol Behav Assess. 2010Mar;32:68–82.

Zinbarg RE, Barlow DH. Structure of anxiety and the anxiety disorders: a hierarchical model. J Abnorm Psychol. 1996May;105(2):181.

Article   CAS   PubMed   Google Scholar  

Campbell-Sills L, Barlow DH, Brown TA, Hofmann SG. Acceptability and suppression of negative emotion in anxiety and mood disorders. Emotion. 2006Nov;6(4):587.

Campbell-Sills L, Barlow DH, Brown TA, Hofmann SG. Effects of suppression and acceptance on emotional responses of individuals with anxiety and mood disorders. Behav Res Ther. 2006Sep 1;44(9):1251–63.

Kashdan TB, Steger MF. Expanding the topography of social anxiety: An experience-sampling assessment of positive emotions, positive events, and emotion suppression. Psychol Sci. 2006Feb;17(2):120–8.

Kashdan TB, Barrios V, Forsyth JP, Steger MF. Experiential avoidance as a generalized psychological vulnerability: Comparisons with coping and emotion regulation strategies. Behav Res Ther. 2006Sep 1;44(9):1301–20.

Moore R, Gillanders D, Stuart S. The impact of group emotion regulation interventions on emotion regulation ability: A systematic review. J Clin Med. 2022Apr 29;11(9):2519.

Forkmann T, Scherer A, Pawelzik M, Mainz V, Drueke B, Boecker M, Gauggel S. Does cognitive behavior therapy alter emotion regulation in inpatients with a depressive disorder? Psychol Res Behav Manag. 2014May;12:147–53.

Papa A, Boland M, Sewell MT. Emotion regulation and CBT. In: O'Donohue WT, Fisher JE, editors. Cognitive behavior therapy: Core principles for practice. Wiley. 2012. p. 273–323. https://doi.org/10.1002/9781118470886.ch11 .

Hayes SC. Acceptance and commitment therapy, relational frame theory, and the third wave of behavioral and cognitive therapies. Behav Ther. 2004Sep 1;35(4):639–65.

Linehan M. Cognitive-behavioral treatment of borderline personality disorder. Hoboken: Guilford press; 1993.

Neacsiu AD, Eberle JW, Kramer R, Wiesmann T, Linehan MM. Dialectical behavior therapy skills for transdiagnostic emotion dysregulation: A pilot randomized controlled trial. Behav Res Ther. 2014Aug;1(59):40–51.

Roemer L, Orsillo SM, Salters-Pedneault K. Efficacy of an acceptance-based behavior therapy for generalized anxiety disorder: evaluation in a randomized controlled trial. J Consult Clin Psychol. 2008Dec;76(6):1083.

Renna ME, Quintero JM, Fresco DM, Mennin DS. Emotion regulation therapy: a mechanism-targeted treatment for disorders of distress. Front Psychol. 2017Feb;6(8): 211118.

Bullis JR, Sauer-Zavala S, Bentley KH, Thompson-Hollands J, Carl JR, Barlow DH. The unified protocol for transdiagnostic treatment of emotional disorders: Preliminary exploration of effectiveness for group delivery. Behav Modif. 2015Mar;39(2):295–321.

Gratz KL, Gunderson JG. Preliminary data on an acceptance-based emotion regulation group intervention for deliberate self-harm among women with borderline personality disorder. Behav Ther. 2006Mar 1;37(1):25–35.

Berking M, Wupperman P, Reichardt A, Pejic T, Dippel A, Znoj H. Emotion-regulation skills as a treatment target in psychotherapy. Behav Res Ther. 2008Nov 1;46(11):1230–7.

Berking M, Whitley B. Emotion Regulation: Definition and Relevance for Mental Health. In: Affect Regulation Training. New York: Springer; 2014. https://doi.org/10.1007/978-1-4939-1022-9_2 .

Hall K, Simpson A, O’donnell R, Sloan E, Staiger PK, Morton J, Ryan D, Nunn B, Best D, Lubman DI. Emotional dysregulation as a target in the treatment of co‐existing substance use and borderline personality disorders: A pilot study. Clin Psychol. 2018;22(2):112–25.

Safer DL, Jo B. Outcome from a randomized controlled trial of group therapy for binge eating disorder: Comparing dialectical behavior therapy adapted for binge eating to an active comparison group therapy. Behav Ther. 2010;41(1):106–20.

Keuthen NJ, Rothbaum BO, Fama J, Altenburger E, Falkenstein MJ, Sprich SE, Kearns M, Meunier S, Jenike MA, Welch SS. DBT-enhanced cognitive-behavioral treatment for trichotillomania: A randomized controlled trial. J Behav Addict. 2012;1:106–14.

Gratz KL, Dixon-Gordon KL, Tull MT. Predictors of treatment response to an adjunctive emotion regulation group therapy for deliberate self-harm among women with borderline personality disorder. Personal Disord Theory Res Treat. 2014;5:97–107.

Barlow DH, Levitt JT, Bufka LF. The dissemination of empirically supported treatments: a view to the future. Behav Res Ther. 1999Jul;1(37):S147–62.

Wang Y, Vlemincx E, Vantieghem I, Dhar M, Dong D, Vandekerckhove M. Bottom-up and cognitive top-down emotion regulation: Experiential emotion regulation and cognitive reappraisal on stress relief and follow-up sleep physiology. Int J Environ Res Public Health. 2022Jun 22;19(13):7621.

Viviani R. Emotion regulation, attention to emotion, and the ventral attentional network. Front Hum Neurosci. 2013Nov;7(7):746.

PubMed   PubMed Central   Google Scholar  

McRae K, Misra S, Prasad AK, Pereira SC, Gross JJ. Bottom-up and top-down emotion generation: implications for emotion regulation. Social cognitive and affective neuroscience. 2012Mar 1;7(3):253–62.

Seligman ME. Phobias and preparedness. Behav Ther. 1971Jul 1;2(3):307–20.

Ochsner KN, Gross JJ. The neural architecture of emotion regulation. Handbook of emotion regulation. 2007;1(1):87–109.

Smits JA, Julian K, Rosenfield D, Powers MB. Threat reappraisal as a mediator of symptom change in cognitive-behavioral treatment of anxiety disorders: a systematic review. J Consult Clin Psychol. 2012Aug;80(4):624.

Bishop SJ. Trait anxiety and impoverished prefrontal control of attention. Nat Neurosci. 2009Jan;12(1):92–8.

Blair KS, Geraci M, Smith BW, Hollon N, DeVido J, Otero M, Blair JR, Pine DS. Reduced dorsal anterior cingulate cortical activity during emotional regulation and top-down attentional control in generalized social phobia, generalized anxiety disorder, and comorbid generalized social phobia/generalized anxiety disorder. Biol Psychiat. 2012Sep 15;72(6):476–82.

Lyubomirsky S, Kasri F, Zehm K. Dysphoric rumination impairs concentration on academic tasks. Cogn Ther Res. 2003Jun;27:309–30.

Johnstone T, Van Reekum CM, Urry HL, Kalin NH, Davidson RJ. Failure to regulate: counterproductive recruitment of top-down prefrontal-subcortical circuitry in major depression. J Neurosci. 2007Aug 15;27(33):8877–84.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Arntz A, De Groot C, Kindt M. Emotional memory is perceptual. J Behav Ther Exp Psychiatry. 2005Mar 1;36(1):19–34.

Gruber H, Oepen R. Emotion regulation strategies and effects in art-making: A narrative synthesis. Arts Psychother. 2018Jul;1(59):65–74.

Abbing A, Baars EW, De Sonneville L. The effectiveness of art therapy for anxiety in adult women: a randomized controlled trial. Front Psychol. 2019May;29(10): 436010.

Haeyen S, van Hooren S, Dehue F, Hutschemaekers G. Development of an art-therapy intervention for patients with personality disorders: an intervention mapping study. International Journal of Art Therapy. 2018Jul 3;23(3):125–35.

Nan JK, Ho RT. Effects of clay art therapy on adults outpatients with major depressive disorder: A randomized controlled trial. J Affect Disord. 2017Aug;1(217):237–45.

Mennin DS, Fresco DM. Advancing emotion regulation perspectives on psychopathology: The challenge of distress disorders. Psychol Inq. 2015Jan 2;26(1):80–92.

Peters GJ. A practical guide to effective behavior change: how to identify what to change in the first place. European Health Psychologist. 2014Oct 1;16(5):142–55.

Ochsner KN, Bunge SA, Gross JJ, Gabrieli JD. Rethinking feelings: an FMRI study of the cognitive regulation of emotion. J Cogn Neurosci. 2002Nov 15;14(8):1215–29.

Ochsner KN, Gross JJ. The cognitive control of emotion. Trends Cogn Sci. 2005May 1;9(5):242–9.

Thiruchselvam R, Blechert J, Sheppes G, Rydstrom A, Gross JJ. The temporal dynamics of emotion regulation: An EEG study of distraction and reappraisal. Biol Psychol. 2011Apr 1;87(1):84–92.

Forkosh J, Drake JE. Coloring versus drawing: Effects of cognitive demand on mood repair, flow, and enjoyment. Art Ther. 2017Apr 3;34(2):75–82.

Gerin W, Davidson KW, Christenfeld NJ, Goyal T, Schwartz JE. The role of angry rumination and distraction in blood pressure recovery from emotional arousal. Psychosom Med. 2006Jan 1;68(1):64–72.

Urry HL, Gross JJ. Emotion regulation in older age. Curr Dir Psychol Sci. 2010Dec;19(6):352–7.

Fresco DM, Moore MT, van Dulmen MH, Segal ZV, Ma SH, Teasdale JD, Williams JM. Initial psychometric properties of the experiences questionnaire: validation of a self-report measure of decentering. Behav Ther. 2007Sep 1;38(3):234–46.

Kross E, Ayduk O. Facilitating adaptive emotional analysis: Distinguishing distanced-analysis of depressive experiences from immersed-analysis and distraction. Pers Soc Psychol Bull. 2008Jul;34(7):924–38.

Ayduk Ö, Kross E. From a distance: implications of spontaneous self-distancing for adaptive self-reflection. J Pers Soc Psychol. 2010May;98(5):809.

Feldman GC, Joormann J, Johnson SL. Responses to positive affect: A self-report measure of rumination and dampening. Cogn Ther Res. 2008Aug;32:507–25.

Gentzler AL, Ramsey MA, Yuen Yi C, Palmer CA, Morey JN. Young adolescents’ emotional and regulatory responses to positive life events: Investigating temperament, attachment, and event characteristics. J Posit Psychol. 2014Mar 4;9(2):108–21.

Harding KA, Hudson MR, Mezulis A. Cognitive mechanisms linking low trait positive affect to depressive symptoms: A prospective diary study. Cogn Emot. 2014Nov 17;28(8):1502–11.

Mennin DS, Fresco DM. Emotion regulation therapy. Handbook of emotion regulation. 2014;2:469–90.

Levitt JT, Brown TA, Orsillo SM, Barlow DH. The effects of acceptance versus suppression of emotion on subjective and psychophysiological response to carbon dioxide challenge in patients with panic disorder. Behav Ther. 2004Sep 1;35(4):747–66.

Wolgast M, Lundh LG, Viborg G. Cognitive reappraisal and acceptance: An experimental comparison of two emotion regulation strategies. Behav Res Ther. 2011Dec 1;49(12):858–66.

Moon BL. The role of metaphor in art therapy: Theory, method, and experience. Charles C Thomas Publisher; 2007.

Kimport ER, Robbins SJ. Efficacy of creative clay work for reducing negative mood: A randomized controlled trial. Art Ther. 2012Jun 1;29(2):74–9.

Drake JE, Hodge A. Drawing versus writing: The role of preference in regulating short-term affect. Art Ther. 2015Jan 2;32(1):27–33.

De Petrillo L, Winner E. Does Art Improve Mood? A Test of a Key Assumption Underlying Art Therapy. Art Ther. 2005;22(4):205–12. https://doi.org/10.1080/07421656.2005.10129521 .

Gross JJ. Emotion regulation: Current status and future prospects. Psychol Inq. 2015Jan 2;26(1):1–26.

Schutte NS, Manes RR, Malouff JM. Antecedent-focused emotion regulation, response modulation and well-being. Curr Psychol. 2009Mar;28:21–31.

Sheppes G, Gross JJ. Emotion regulation effectiveness: What works when. Handb Psychol. 2012Sep;26(5):391–406.

Aldao A, Nolen-Hoeksema S. One versus many: Capturing the use of multiple emotion regulation strategies in response to an emotion-eliciting stimulus. Cogn Emot. 2013Jun 1;27(4):753–60.

Lee M, Choi H. Art-based emotion regulation in major depression: Framework for intervention. Arts Psychother. 2023Mar;24: 102018.

Gohm CL, Clore GL. Four latent traits of emotional experience and their involvement in well-being, coping, and attributional style. Cogn Emot. 2002Jul 1;16(4):495–518.

Tamir M, Vishkin A, Gutentag T. Emotion regulation is motivated. Emotion. 2020Feb;20(1):115.

Nauphal M, Curreri AJ, Cardona ND, Meyer ER, Southward MW, Sauer-Zavala S. Measuring emotion regulation skill use during treatment: A promising methodological approach. Assessment. 2023Apr;30(3):592–605.

Aldao A, Nolen-Hoeksema S, Schweizer S. Emotion-regulation strategies across psychopathology: A meta-analytic review. Clin Psychol Rev. 2010Mar 1;30(2):217–37.

Bradley MM, Lang PJ. Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry. 1994Mar 1;25(1):49–59.

Download references

Acknowledgements

The present researchers express their gratitude to the Kangdong Sacred Heart Hospital for its help and support in this research. Appreciation is also extended to all participating patients, clinicians, health care professionals, and the advisory board in all steps of the development. There are no individuals or funding organizations, other than the co-authors, who contributed directly or indirectly to this article.

Not applicable.

Author information

Authors and affiliations.

Department of Psychiatry, Kangdong Sacred Heart Hospital, 150, Seongan-Ro, Gangdong-Gu, Seoul, Republic of Korea

Myungjoo Lee & Young Tak Jo

Department of Bio-Medical Engineering, Ajou University, 206, World Cup-Ro, Yeongtong-Gu, Gyeonggi-do, Republic of Korea

You can also search for this author in PubMed   Google Scholar

Contributions

ML contributed to plan and design the study with support from the rest of the study team. YT registered the trial. ML collected, and analyzed participant data. ML drafted and edited the manuscript. All authors reviewed and/or approved the final manuscript for submission.

Corresponding author

Correspondence to Young Tak Jo .

Ethics declarations

Ethics approval and consent to participate.

The ethical approvals for the current research were obtained from the Institutional Review Board of Kangdong Sacred Heart Hospital, PAR2 (IRB no. 2023–12-002), and PAR3 (no. 2024–02-019). In PAR2, informed consent was exempted due to its retrospective nature. Nevertheless, all patients who participated in the therapeutic sessions were requested to sign a consent form for later use of their artwork for educational and research purposes. In PAR 3, informed consent was obtained from the healthcare professionals, including physicians and nurses, who were involved in the program.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Lee, M., Choi, H. & Jo, Y.T. Targeting emotion dysregulation in depression: an intervention mapping protocol augmented by participatory action research. BMC Psychiatry 24 , 595 (2024). https://doi.org/10.1186/s12888-024-06045-y

Download citation

Received : 05 April 2024

Accepted : 23 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1186/s12888-024-06045-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Psychotherapy

BMC Psychiatry

ISSN: 1471-244X

• General enquiries: [email protected]