systematic approach to literature review

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

• the definition of a systematic literature review
• the purpose of a systematic literature review
• the different types of systematic reviews
• how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

• Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
• Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
• Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
• Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
• Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
• Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
• Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
• Prognostic reviews examine the course of a disease and its potential outcomes.
• Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
• Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

• Decide on your team.
• Formulate your question.
• Plan your research protocol.
• Search for the literature.
• Screen the literature.
• Assess the quality of the studies.
• Extract the data.
• Analyze the results.
• Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

• Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
• Intervention : which main intervention or treatment are you considering for assessment?
• Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
• Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

• the objectives of your project
• the specific methods and processes that you’ll use
• the eligibility criteria of the individual studies
• how you plan to extract data from individual studies
• which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

• non-peer-reviewed journals
• pharmaceutical industry files
• conference proceedings
• pharmaceutical company websites
• internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

• The databases you search and which years they cover
• The dates you first run the searches, and when they’re updated
• Which strategies you use, including search terms
• The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

• Pre-screening of all titles and abstracts, and selecting those appropriate
• Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

• Were the participants really randomly allocated to their groups?
• Were the groups similar in terms of prognostic factors?
• Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

• Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
• Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
• Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

• Find My Rep

You are here

Sage online ordering services are currently unavailable due to network disruption. We are working to restore service as soon as possible. In the meantime, our books are still available widely via retailers and online. Currently we are only able to supply digital inspection copies. These can be requested via this site, or direct from Vitalsource and Kortext . If you need a print copy, please contact your local Academic Sales Consultant, who will be able to arrange to send this once our systems are back online.

I f you need further assistance please visit our Contact us page for further information.  Thank you for your patience and we apologise for the inconvenience.

Systematic Approaches to a Successful Literature Review

• Andrew Booth - The University of Sheffield, UK
• Anthea Sutton - The University of Sheffield, UK
• Mark Clowes - Sheffield University, UK
• Marrissa Martyn-St James - Sheffield University, UK
• Description

The perfect project support for any social sciences student, this edition also includes a new chapter on analysing mixed methods research.

Supplements

Student Resources (Free to access) A literature review starter template to demonstrate the sections you need to include for a successful written review. A source credibility checklist to help you assess and think critically about the sources you choose. A source tracker template to help you keep track of your sources and know what you need to include in your audit trail. A downloadable exercise workbook and suggested answers.  A collated list of tried-and-tested tools , including freely available technologies to help you search databases efficiently, plan your work, and keep track of references. A project diary template and example. A complete glossary of terms. Instructor Resources (Log-in needed) PowerPoint slide templates including 10-15 slides per chapter, which can be downloaded and customized for use in your own presentations.

The Systematic Approaches to a Successful Literature Review (third edition) by Andrew Booth, Anthea Sutton, Mark Clowes and Marrissa Martyn-St James is a comprehensive overview of the entire evidence synthesis process – from selecting the appropriate method for an evidence synthesis topic all the way to the analysis and dissemination of the review. This book is of relevance to anyone interested in evidence synthesis – from trainees to researchers to decision-makers. Anyone can learn something from this book, whether you are a beginner, intermediate, or advance researcher in evidence synthesis. This book is perfect for university-level courses or for anyone interested in evidence synthesis. The exercises, toolbox, key learning points, and frequently asked questions were particularly helpful in advancing my learning.

For our masters level students doing their literature review dissertation this provides effective guidance in approaching their work in a systematic fashion.

Great resource. Easy to read, with helpful tables and diagrams that catch the students' attention and they find easy to recall. The examples and up-to-date links to external sources are also invaluable springboards for the students.

Post-COVID many more students prefer to use the electronic versions of books and the library is also keen to adopt more books in this format, so this is very helpful to enable the maximum number of students to access the helpful text with easy to follow guidance .

I have put this at the top of my reading list for my module on evidence based practice which is like a mini dissertation for level 5 students. It is a comprehensive read and sets out the stages involved in a successful literature review. My students have a problem with this part of the module and this book is a godsend

This book is very important for students to understand how to do an in-depth literature review as a support and motivation for their research.

I did not receive an inspection copy to use

Good comprehensive text - easy to follow

Very clear and useful

Preview this book

For instructors, select a purchasing option.

• Electronic Order Options VitalSource Amazon Kindle Google Play eBooks.com Kobo

• A-Z Publications

Annual Review of Psychology

Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.

• Andy P. Siddaway 1 , Alex M. Wood 2 , and Larry V. Hedges 3
• View Affiliations Hide Affiliations Affiliations: 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected] 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected]
• Vol. 70:747-770 (Volume publication date January 2019) https://doi.org/10.1146/annurev-psych-010418-102803
• First published as a Review in Advance on August 08, 2018
• Copyright © 2019 by Annual Reviews. All rights reserved

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Article metrics loading...

Full text loading...

Literature Cited

• APA Publ. Commun. Board Work. Group J. Artic. Rep. Stand. 2008 . Reporting standards for research in psychology: Why do we need them? What might they be?. Am. Psychol . 63 : 848– 49 [Google Scholar]
• Baumeister RF 2013 . Writing a literature review. The Portable Mentor: Expert Guide to a Successful Career in Psychology MJ Prinstein, MD Patterson 119– 32 New York: Springer, 2nd ed.. [Google Scholar]
• Baumeister RF , Leary MR 1995 . The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychol. Bull. 117 : 497– 529 [Google Scholar]
• Baumeister RF , Leary MR 1997 . Writing narrative literature reviews. Rev. Gen. Psychol. 3 : 311– 20 Presents a thorough and thoughtful guide to conducting narrative reviews. [Google Scholar]
• Bem DJ 1995 . Writing a review article for Psychological Bulletin. Psychol . Bull 118 : 172– 77 [Google Scholar]
• Borenstein M , Hedges LV , Higgins JPT , Rothstein HR 2009 . Introduction to Meta-Analysis New York: Wiley Presents a comprehensive introduction to meta-analysis. [Google Scholar]
• Borenstein M , Higgins JPT , Hedges LV , Rothstein HR 2017 . Basics of meta-analysis: I 2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8 : 5– 18 [Google Scholar]
• Braver SL , Thoemmes FJ , Rosenthal R 2014 . Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9 : 333– 42 [Google Scholar]
• Bushman BJ 1994 . Vote-counting procedures. The Handbook of Research Synthesis H Cooper, LV Hedges 193– 214 New York: Russell Sage Found. [Google Scholar]
• Cesario J 2014 . Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9 : 40– 48 [Google Scholar]
• Chalmers I 2007 . The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the importance of systematic reviews. Treating Individuals: From Randomised Trials to Personalised Medicine PM Rothwell 37– 58 London: Lancet [Google Scholar]
• Cochrane Collab. 2003 . Glossary Rep., Cochrane Collab. London: http://community.cochrane.org/glossary Presents a comprehensive glossary of terms relevant to systematic reviews. [Google Scholar]
• Cohn LD , Becker BJ 2003 . How meta-analysis increases statistical power. Psychol. Methods 8 : 243– 53 [Google Scholar]
• Cooper HM 2003 . Editorial. Psychol. Bull. 129 : 3– 9 [Google Scholar]
• Cooper HM 2016 . Research Synthesis and Meta-Analysis: A Step-by-Step Approach Thousand Oaks, CA: Sage, 5th ed.. Presents a comprehensive introduction to research synthesis and meta-analysis. [Google Scholar]
• Cooper HM , Hedges LV , Valentine JC 2009 . The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Cumming G 2014 . The new statistics: why and how. Psychol. Sci. 25 : 7– 29 Discusses the limitations of null hypothesis significance testing and viable alternative approaches. [Google Scholar]
• Earp BD , Trafimow D 2015 . Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6 : 621 [Google Scholar]
• Etz A , Vandekerckhove J 2016 . A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11 : e0149794 [Google Scholar]
• Ferguson CJ , Brannick MT 2012 . Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17 : 120– 28 [Google Scholar]
• Fleiss JL , Berlin JA 2009 . Effect sizes for dichotomous data. The Handbook of Research Synthesis and Meta-Analysis H Cooper, LV Hedges, JC Valentine 237– 53 New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Garside R 2014 . Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how. Innovation 27 : 67– 79 [Google Scholar]
• Hedges LV , Olkin I 1980 . Vote count methods in research synthesis. Psychol. Bull. 88 : 359– 69 [Google Scholar]
• Hedges LV , Pigott TD 2001 . The power of statistical tests in meta-analysis. Psychol. Methods 6 : 203– 17 [Google Scholar]
• Higgins JPT , Green S 2011 . Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 London: Cochrane Collab. Presents comprehensive and regularly updated guidelines on systematic reviews. [Google Scholar]
• John LK , Loewenstein G , Prelec D 2012 . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 : 524– 32 [Google Scholar]
• Juni P , Witschi A , Bloch R , Egger M 1999 . The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 : 1054– 60 [Google Scholar]
• Klein O , Doyen S , Leys C , Magalhães de Saldanha da Gama PA , Miller S et al. 2012 . Low hopes, high expectations: expectancy effects and the replicability of behavioral experiments. Perspect. Psychol. Sci. 7 : 6 572– 84 [Google Scholar]
• Lau J , Antman EM , Jimenez-Silva J , Kupelnick B , Mosteller F , Chalmers TC 1992 . Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med. 327 : 248– 54 [Google Scholar]
• Light RJ , Smith PV 1971 . Accumulating evidence: procedures for resolving contradictions among different research studies. Harvard Educ. Rev. 41 : 429– 71 [Google Scholar]
• Lipsey MW , Wilson D 2001 . Practical Meta-Analysis London: Sage Comprehensive and clear explanation of meta-analysis. [Google Scholar]
• Matt GE , Cook TD 1994 . Threats to the validity of research synthesis. The Handbook of Research Synthesis H Cooper, LV Hedges 503– 20 New York: Russell Sage Found. [Google Scholar]
• Maxwell SE , Lau MY , Howard GS 2015 . Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. Am. Psychol. 70 : 487– 98 [Google Scholar]
• Moher D , Hopewell S , Schulz KF , Montori V , Gøtzsche PC et al. 2010 . CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 : c869 [Google Scholar]
• Moher D , Liberati A , Tetzlaff J , Altman DG PRISMA Group. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339 : 332– 36 Comprehensive reporting guidelines for systematic reviews. [Google Scholar]
• Morrison A , Polisena J , Husereau D , Moulton K , Clark M et al. 2012 . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int. J. Technol. Assess. Health Care 28 : 138– 44 [Google Scholar]
• Nelson LD , Simmons J , Simonsohn U 2018 . Psychology's renaissance. Annu. Rev. Psychol. 69 : 511– 34 [Google Scholar]
• Noblit GW , Hare RD 1988 . Meta-Ethnography: Synthesizing Qualitative Studies Newbury Park, CA: Sage [Google Scholar]
• Olivo SA , Macedo LG , Gadotti IC , Fuentes J , Stanton T , Magee DJ 2008 . Scales to assess the quality of randomized controlled trials: a systematic review. Phys. Ther. 88 : 156– 75 [Google Scholar]
• Open Sci. Collab. 2015 . Estimating the reproducibility of psychological science. Science 349 : 943 [Google Scholar]
• Paterson BL , Thorne SE , Canam C , Jillings C 2001 . Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis Thousand Oaks, CA: Sage [Google Scholar]
• Patil P , Peng RD , Leek JT 2016 . What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 : 539– 44 [Google Scholar]
• Rosenthal R 1979 . The “file drawer problem” and tolerance for null results. Psychol. Bull. 86 : 638– 41 [Google Scholar]
• Rosnow RL , Rosenthal R 1989 . Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44 : 1276– 84 [Google Scholar]
• Sanderson S , Tatt ID , Higgins JP 2007 . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 : 666– 76 [Google Scholar]
• Schreiber R , Crooks D , Stern PN 1997 . Qualitative meta-analysis. Completing a Qualitative Project: Details and Dialogue JM Morse 311– 26 Thousand Oaks, CA: Sage [Google Scholar]
• Shrout PE , Rodgers JL 2018 . Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69 : 487– 510 [Google Scholar]
• Stroebe W , Strack F 2014 . The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9 : 59– 71 [Google Scholar]
• Stroup DF , Berlin JA , Morton SC , Olkin I , Williamson GD et al. 2000 . Meta-analysis of observational studies in epidemiology (MOOSE): a proposal for reporting. JAMA 283 : 2008– 12 [Google Scholar]
• Thorne S , Jensen L , Kearney MH , Noblit G , Sandelowski M 2004 . Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14 : 1342– 65 [Google Scholar]
• Tong A , Flemming K , McInnes E , Oliver S , Craig J 2012 . Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med. Res. Methodol. 12 : 181– 88 [Google Scholar]
• Trickey D , Siddaway AP , Meiser-Stedman R , Serpell L , Field AP 2012 . A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clin. Psychol. Rev. 32 : 122– 38 [Google Scholar]
• Valentine JC , Biglan A , Boruch RF , Castro FG , Collins LM et al. 2011 . Replication in prevention science. Prev. Sci. 12 : 103– 17 [Google Scholar]
• Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved July 5, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

• UNC Libraries
• HSL Academic Process
• Systematic Reviews

Systematic Reviews: Home

Created by health science librarians.

• Systematic review resources

What is a Systematic Review?

A simplified process map, how can the library help, publications by hsl librarians, systematic reviews in non-health disciplines, resources for performing systematic reviews.

• Step 1: Complete Pre-Review Tasks
• Step 2: Develop a Protocol
• Step 3: Conduct Literature Searches
• Step 4: Manage Citations
• Step 5: Screen Citations
• Step 6: Assess Quality of Included Studies
• Step 7: Extract Data from Included Studies
• Step 8: Write the Review

  Check our FAQ's

   Email us

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

A systematic review is a literature review that gathers all of the available evidence matching pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods, documented in a protocol, to minimize bias , provide reliable findings , and inform decision-making.  ¹  

There are many types of literature reviews.

Before beginning a systematic review, consider whether it is the best type of review for your question, goals, and resources. The table below compares a few different types of reviews to help you decide which is best for you. 

• Scoping Review Guide For more information about scoping reviews, refer to the UNC HSL Scoping Review Guide.

• UNC HSL's Simplified, Step-by-Step Process Map A PDF file of the HSL's Systematic Review Process Map.
• Text-Only: UNC HSL's Systematic Reviews - A Simplified, Step-by-Step Process A text-only PDF file of HSL's Systematic Review Process Map.

The average systematic review takes 1,168 hours to complete. ¹   A librarian can help you speed up the process.

Systematic reviews follow established guidelines and best practices to produce high-quality research. Librarian involvement in systematic reviews is based on two levels. In Tier 1, your research team can consult with the librarian as needed. The librarian will answer questions and give you recommendations for tools to use. In Tier 2, the librarian will be an active member of your research team and co-author on your review. Roles and expectations of librarians vary based on the level of involvement desired. Examples of these differences are outlined in the table below.

• Request a systematic or scoping review consultation

The following are systematic and scoping reviews co-authored by HSL librarians.

Only the most recent 15 results are listed. Click the website link at the bottom of the list to see all reviews co-authored by HSL librarians in PubMed

Researchers conduct systematic reviews in a variety of disciplines.  If your focus is on a topic outside of the health sciences, you may want to also consult the resources below to learn how systematic reviews may vary in your field.  You can also contact a librarian for your discipline with questions.

• EPPI-Centre methods for conducting systematic reviews The EPPI-Centre develops methods and tools for conducting systematic reviews, including reviews for education, public and social policy.

Environmental Topics

• Collaboration for Environmental Evidence (CEE) CEE seeks to promote and deliver evidence syntheses on issues of greatest concern to environmental policy and practice as a public service

Social Sciences

• Siddaway AP, Wood AM, Hedges LV. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annu Rev Psychol. 2019 Jan 4;70:747-770. doi: 10.1146/annurev-psych-010418-102803. A resource for psychology systematic reviews, which also covers qualitative meta-syntheses or meta-ethnographies
• The Campbell Collaboration

Social Work

Software engineering

• Guidelines for Performing Systematic Literature Reviews in Software Engineering The objective of this report is to propose comprehensive guidelines for systematic literature reviews appropriate for software engineering researchers, including PhD students.

Sport, Exercise, & Nutrition

• Application of systematic review methodology to the field of nutrition by Tufts Evidence-based Practice Center Publication Date: 2009
• Systematic Reviews and Meta-Analysis — Open & Free (Open Learning Initiative) The course follows guidelines and standards developed by the Campbell Collaboration, based on empirical evidence about how to produce the most comprehensive and accurate reviews of research

• Systematic Reviews by David Gough, Sandy Oliver & James Thomas Publication Date: 2020

Updating reviews

• Updating systematic reviews by University of Ottawa Evidence-based Practice Center Publication Date: 2007
• Next: Step 1: Complete Pre-Review Tasks >>
• Last Updated: May 16, 2024 3:24 PM
• URL: https://guides.lib.unc.edu/systematic-reviews

What is a Systematic Literature Review?

A systematic literature review (SLR) is an independent academic method that aims to identify and evaluate all relevant literature on a topic in order to derive conclusions about the question under consideration. "Systematic reviews are undertaken to clarify the state of existing research and the implications that should be drawn from this." (Feak & Swales, 2009, p. 3) An SLR can demonstrate the current state of research on a topic, while identifying gaps and areas requiring further research with regard to a given research question. A formal methodological approach is pursued in order to reduce distortions caused by an overly restrictive selection of the available literature and to increase the reliability of the literature selected (Tranfield, Denyer & Smart, 2003). A special aspect in this regard is the fact that a research objective is defined for the search itself and the criteria for determining what is to be included and excluded are defined prior to conducting the search. The search is mainly performed in electronic literature databases (such as Business Source Complete or Web of Science), but also includes manual searches (reviews of reference lists in relevant sources) and the identification of literature not yet published in order to obtain a comprehensive overview of a research topic.

An SLR protocol documents all the information gathered and the steps taken as part of an SLR in order to make the selection process transparent and reproducible. The PRISMA flow-diagram support you in making the selection process visible.

In an ideal scenario, experts from the respective research discipline, as well as experts working in the relevant field and in libraries, should be involved in setting the search terms . As a rule, the literature is selected by two or more reviewers working independently of one another. Both measures serve the purpose of increasing the objectivity of the literature selection. An SLR must, then, be more than merely a summary of a topic (Briner & Denyer, 2012). As such, it also distinguishes itself from “ordinary” surveys of the available literature. The following table shows the differences between an SLR and an “ordinary” literature review.

• Charts of BSWL workshop (pdf, 2.88 MB)
• Listen to the interview (mp4, 12.35 MB)

Differences to "common" literature reviews

What are the objectives of SLRs?

• Avoidance of research redundancies despite a growing amount of publications
• Identification of research areas, gaps and methods
• Input for evidence-based management, which allows to base management decisions on scientific methods and findings
• Identification of links between different areas of researc

Process steps of an SLR

A SLR has several process steps which are defined differently in the literature (Fink 2014, p. 4; Guba 2008, Transfield et al. 2003). We distinguish the following steps which are adapted to the economics and management research area:

1. Defining research questions

Briner & Denyer (2009, p. 347ff.) have developed the CIMO scheme to establish clearly formulated and answerable research questions in the field of economic sciences:

C – CONTEXT:  Which individuals, relationships, institutional frameworks and systems are being investigated?

I – Intervention:  The effects of which event, action or activity are being investigated?

M – Mechanisms:  Which mechanisms can explain the relationship between interventions and results? Under what conditions do these mechanisms take effect?

O – Outcomes:  What are the effects of the intervention? How are the results measured? What are intended and unintended effects?

The objective of the systematic literature review is used to formulate research questions such as “How can a project team be led effectively?”. Since there are numerous interpretations and constructs for “effective”, “leadership” and “project team”, these terms must be particularized.

With the aid of the scheme, the following concrete research questions can be derived with regard to this example:

Under what conditions (C) does leadership style (I) influence the performance of project teams (O)?

Which constructs have an effect upon the influence of leadership style (I) on a project team’s performance (O)?          

Research questions do not necessarily need to follow the CIMO scheme, but they should:

• ... be formulated in a clear, focused and comprehensible manner and be answerable;
• ... have been determined prior to carrying out the SLR;
• ... consist of general and specific questions.

As early as this stage, the criteria for inclusion and exclusion are also defined. The selection of the criteria must be well-grounded. This may include conceptual factors such as a geographical or temporal restrictions, congruent definitions of constructs, as well as quality criteria (journal impact factor > x).

2. Selecting databases and other research sources

The selection of sources must be described and explained in detail. The aim is to find a balance between the relevance of the sources (content-related fit) and the scope of the sources.

In the field of economic sciences, there are a number of literature databases that can be searched as part of an SLR. Some examples in this regard are:

• Business Source Complete
• ProQuest One Business
• EconBiz        

Our video " Selecting the right databases " explains how to find relevant databases for your topic.

Literature databases are an important source of research for SLRs, as they can minimize distortions caused by an individual literature selection (selection bias), while offering advantages for a systematic search due to their data structure. The aim is to find all database entries on a topic and thus keep the retrieval bias low (tutorial on retrieval bias ).  Besides articles from scientific journals, it is important to inlcude working papers, conference proceedings, etc to reduce the publication bias ( tutorial on publication bias ).

Our online self-study course " Searching economic databases " explains step 2 und 3.

3. Defining search terms

Once the literature databases and other research sources have been selected, search terms are defined. For this purpose, the research topic/questions is/are divided into blocks of terms of equal ranking. This approach is called the block-building method (Guba 2008, p. 63). The so-called document-term matrix, which lists topic blocks and search terms according to a scheme, is helpful in this regard. The aim is to identify as many different synonyms as possible for the partial terms. A precisely formulated research question facilitates the identification of relevant search terms. In addition, keywords from particularly relevant articles support the formulation of search terms.

A document-term matrix for the topic “The influence of management style on the performance of project teams” is shown in this example .

Identification of headwords and keywords

When setting search terms, a distinction must be made between subject headings and keywords, both of which are described below:

• appear in the title, abstract and/or text
• sometimes specified by the author, but in most cases automatically generated
• non-standardized
• different spellings and forms (singular/plural) must be searched separately

Subject headings

• describe the content
• are generated by an editorial team
• are listed in a standardized list (thesaurus)
• may comprise various keywords
• include different spellings
• database-specific

Subject headings are a standardized list of words that are generated by the specialists in charge of some databases. This so-called index of subject headings (thesaurus) helps searchers find relevant articles, since the headwords indicate the content of a publication. By contrast, an ordinary keyword search does not necessarily result in a content-related fit, since the database also displays articles in which, for example, a word appears once in the abstract, even though the article’s content does not cover the topic.

Nevertheless, searches using both headwords and keywords should be conducted, since some articles may not yet have been assigned headwords, or errors may have occurred during the assignment of headwords. 

To add headwords to your search in the Business Source Complete database, please select the Thesaurus tab at the top. Here you can find headwords in a new search field and integrate them into your search query. In the search history, headwords are marked with the addition DE (descriptor).

The EconBiz database of the German National Library of Economics (ZBW – Leibniz Information Centre for Economics), which also contains German-language literature, has created its own index of subject headings with the STW Thesaurus for Economics . Headwords are integrated into the search by being used in the search query.

Since the indexes of subject headings divide terms into synonyms, generic terms and sub-aspects, they facilitate the creation of a document-term matrix. For this purpose it is advisable to specify in the document-term matrix the origin of the search terms (STW Thesaurus for Economics, Business Source Complete, etc.).

Searching in literature databases

Once the document-term matrix has been defined, the search in literature databases begins. It is recommended to enter each word of the document-term matrix individually into the database in order to obtain a good overview of the number of hits per word. Finally, all the words contained in a block of terms are linked with the Boolean operator OR and thereby a union of all the words is formed. The latter are then linked with each other using the Boolean operator AND. In doing so, each block should be added individually in order to see to what degree the number of hits decreases.

Since the search query must be set up separately for each database, tools such as  LitSonar  have been developed to enable a systematic search across different databases. LitSonar was created by  Professor Dr. Ali Sunyaev (Institute of Applied Informatics and Formal Description Methods – AIFB) at the Karlsruhe Institute of Technology.

Advanced search

Certain database-specific commands can be used to refine a search, for example, by taking variable word endings into account (*) or specifying the distance between two words, etc. Our overview shows the most important search commands for our top databases.

Additional searches in sources other than literature databases

In addition to literature databases, other sources should also be searched. Fink (2014, p. 27) lists the following reasons for this:

• the topic is new and not yet included in indexes of subject headings;
• search terms are not used congruently in articles because uniform definitions do not exist;
• some studies are still in the process of being published, or have been completed, but not published.

Therefore, further search strategies are manual search, bibliographic analysis, personal contacts and academic networks (Briner & Denyer, p. 349). Manual search means that you go through the source information of relevant articles and supplement your hit list accordingly. In addition, you should conduct a targeted search for so-called gray literature, that is, literature not distributed via the book trade, such as working papers from specialist areas and conference reports. By including different types of publications, the so-called publication bias (DBWM video “Understanding publication bias” ) – that is, distortions due to exclusive use of articles from peer-reviewed journals – should be kept to a minimum.

The PRESS-Checklist can support you to check the correctness of your search terms.

4. Merging hits from different databases

In principle, large amounts of data can be easily collected, structured and sorted with data processing programs such as Excel. Another option is to use reference management programs such as EndNote, Citavi or Zotero. The Saxon State and University Library Dresden (SLUB Dresden) provides an  overview of current reference management programs  . Software for qualitative data analysis such as NVivo is equally suited for data processing. A comprehensive overview of the features of different tools that support the SLR process can be found in Bandara et al. (2015).

Our online-self study course "Managing literature with Citavi" shows you how to use the reference management software Citavi.

When conducting an SLR, you should specify for each hit the database from which it originates and the date on which the query was made. In addition, you should always indicate how many hits you have identified in the various databases or, for example, by manual search.

Exporting data from literature databases

Exporting from literature databases is very easy. In  Business Source Complete  , you must first click on the “Share” button in the hit list, then “Email a link to download exported results” at the very bottom and then select the appropriate format for the respective literature program.

Exporting data from the literature database  EconBiz  is somewhat more complex. Here you must first create a marked list and then select each hit individually and add it to the marked list. Afterwards, articles on the list can be exported.

After merging all hits from the various databases, duplicate entries (duplicates) are deleted.

5. Applying inclusion and exclusion criteria

All publications are evaluated in the literature management program applying the previously defined criteria for inclusion and exclusion. Only those sources that survive this selection process will subsequently be analyzed. The review process and inclusion criteria should be tested with a small sample and adjustments made if necessary before applying it to all articles. In the ideal case, even this selection would be carried out by more than one person, with each working independently of one another. It needs to be made clear how discrepancies between reviewers are dealt with. 

The review of the criteria for inclusion and exclusion is primarily based on the title, abstract and subject headings in the databases, as well as on the keywords provided by the authors of a publication in the first step. In a second step the whole article / source will be read.

You can create tag words for the inclusion and exclusion in your literature management tool to keep an overview.

In addition to the common literature management tools, you can also use software tools that have been developed to support SLRs. The central library of the university in Zurich has published an overview and evaluation of different tools based on a survey among researchers. --> View SLR tools

The selection process needs to be made transparent. The PRISMA flow diagram supports the visualization of the number of included / excluded studies.

Forward and backward search

Should it become apparent that the number of sources found is relatively small, or if you wish to proceed with particular thoroughness, a forward-and-backward search based on the sources found is recommendable (Webster & Watson 2002, p. xvi). A backward search means going through the bibliographies of the sources found. A forward search, by contrast, identifies articles that have cited the relevant publications. The Web of Science and Scopus databases can be used to perform citation analyses.

6. Perform the review

As the next step, the remaining titles are analyzed as to their content by reading them several times in full. Information is extracted according to defined criteria and the quality of the publications is evaluated. If the data extraction is carried out by more than one person, a training ensures that there will be no differences between the reviewers.

Depending on the research questions there exist diffent methods for data abstraction (content analysis, concept matrix etc.). A so-called concept matrix can be used to structure the content of information (Webster & Watson 2002, p. xvii). The image to the right gives an example of a concept matrix according to Becker (2014).

Particularly in the field of economic sciences, the evaluation of a study’s quality cannot be performed according to a generally valid scheme, such as those existing in the field of medicine, for instance. Quality assessment therefore depends largely on the research questions.

Based on the findings of individual studies, a meta-level is then applied to try to understand what similarities and differences exist between the publications, what research gaps exist, etc. This may also result in the development of a theoretical model or reference framework.

Example concept matrix (Becker 2013) on the topic Business Process Management

7. Synthesizing results

Once the review has been conducted, the results must be compiled and, on the basis of these, conclusions derived with regard to the research question (Fink 2014, p. 199ff.). This includes, for example, the following aspects:

• historical development of topics (histogram, time series: when, and how frequently, did publications on the research topic appear?);
• overview of journals, authors or specialist disciplines dealing with the topic;
• comparison of applied statistical methods;
• topics covered by research;
• identifying research gaps;
• developing a reference framework;
• developing constructs;
• performing a meta-analysis: comparison of the correlations of the results of different empirical studies (see for example Fink 2014, p. 203 on conducting meta-analyses)

Publications about the method

Bandara, W., Furtmueller, E., Miskon, S., Gorbacheva, E., & Beekhuyzen, J. (2015). Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support.  Communications of the Association for Information Systems . 34(8), 154-204.

Booth, A., Papaioannou, D., and Sutton, A. (2012)  Systematic approaches to a successful literature review.  London: Sage.

Briner, R. B., & Denyer, D. (2012). Systematic Review and Evidence Synthesis as a Practice and Scholarship Tool. In Rousseau, D. M. (Hrsg.),  The Oxford Handbook of Evidenence Based Management . (S. 112-129). Oxford: Oxford University Press.

Durach, C. F., Wieland, A., & Machuca, Jose A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review . International Journal of Physical Distribution & Logistic Management , 46 (1/2), 118-137. doi:  https://doi.org/10.1108/IJPDLM-05-2013-0133

Feak, C. B., & Swales, J. M. (2009). Telling a Research Story: Writing a Literature Review.  English in Today's Research World 2.  Ann Arbor: University of Michigan Press. doi:  10.3998/mpub.309338

Fink, A. (2014).  Conducting Research Literature Reviews: From the Internet to Paper  (4. Aufl.). Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.

Fisch, C., & Block, J. (2018). Six tips for your (systematic) literature review in business and management research.  Management Review Quarterly,  68, 103–106 (2018).  doi.org/10.1007/s11301-018-0142-x

Guba, B. (2008). Systematische Literaturrecherche.  Wiener Medizinische Wochenschrift , 158 (1-2), S. 62-69. doi:  doi.org/10.1007/s10354-007-0500-0  Hart, C.  Doing a literature review: releasing the social science research imagination.  London: Sage.

Jesson, J. K., Metheson, L. & Lacey, F. (2011).  Doing your Literature Review - traditional and Systematic Techniques . Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71.

Petticrew, M. and Roberts, H. (2006).  Systematic Reviews in the Social Sciences: A Practical Guide . Oxford:Blackwell. Ridley, D. (2012).  The literature review: A step-by-step guide . 2nd edn. London: Sage. 

Chang, W. and Taylor, S.A. (2016), The Effectiveness of Customer Participation in New Product Development: A Meta-Analysis,  Journal of Marketing , American Marketing Association, Los Angeles, CA, Vol. 80 No. 1, pp. 47–64.

Tranfield, D., Denyer, D. & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review.  British Journal of Management , 14 (3), S. 207-222. doi:  https://doi.org/10.1111/1467-8551.00375

Webster, J., & Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review.  Management Information Systems Quarterly , 26(2), xiii-xxiii.  http://www.jstor.org/stable/4132319

Durach, C. F., Wieland, A. & Machuca, Jose. A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review. International Journal of Physical Distribution & Logistics Management, 45(1/2), 118 – 137.

What is particularly good about this example is that search terms were defined by a number of experts and the review was conducted by three researchers working independently of one another. Furthermore, the search terms used have been very well extracted and the procedure of the literature selection very well described.

On the downside, the restriction to English-language literature brings the language bias into play, even though the authors consider it to be insignificant for the subject area.

Bos-Nehles, A., Renkema, M. & Janssen, M. (2017). HRM and innovative work behaviour: a systematic literature review. Personnel Review, 46(7), pp. 1228-1253

• Only very specific keywords used
• No precise information on how the review process was carried out (who reviewed articles?)
• Only journals with impact factor (publication bias)

Jia, F., Orzes, G., Sartor, M. & Nassimbeni, G. (2017). Global sourcing strategy and structure: towards a conceptual framework. International Journal of Operations & Production Management, 37(7), 840-864

• Research questions are explicitly presented
• Search string very detailed
• Exact description of the review process
• 2 persons conducted the review independently of each other

Franziska Klatt

[email protected]

+49 30 314-29778

Privacy notice: The TU Berlin offers a chat information service. If you enable it, your IP address and chat messages will be transmitted to external EU servers. more information

The chat is currently unavailable.

Please use our alternative contact options.

• Open access
• Published: 14 August 2018

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

• Chris Cooper   ORCID: orcid.org/0000-0003-0864-5607 1 ,
• Andrew Booth 2 ,
• Jo Varley-Campbell 1 ,
• Nicky Britten 3 &
• Ruth Garside 4  

BMC Medical Research Methodology volume  18 , Article number:  85 ( 2018 ) Cite this article

207k Accesses

117 Altmetric

Metrics details

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before.

The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

A literature review.

Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through ‘pearl growing’, citation chasing, a search of PubMed using the systematic review methods filter, and the authors’ topic knowledge.

The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of ‘key stages’ in the process of literature searching.

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

Conclusions

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Peer Review reports

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving review stakeholders clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. This is in contrast to the information science literature, which has developed information processing models as an explicit basis for dialogue and empirical testing. Without an explicit model, research in the process of systematic literature searching will remain immature and potentially uneven, and the development of shared information models will be assumed but never articulated.

One way of developing such a conceptual model is by formally examining the implicit “programme theory” as embodied in key methodological texts. The aim of this review is therefore to determine if a shared model of the literature searching process in systematic reviews can be detected across guidance documents and, if so, how this process is reported and supported.

Identifying guidance

Key texts (henceforth referred to as “guidance”) were identified based upon their accessibility to, and prominence within, United Kingdom systematic reviewing practice. The United Kingdom occupies a prominent position in the science of health information retrieval, as quantified by such objective measures as the authorship of papers, the number of Cochrane groups based in the UK, membership and leadership of groups such as the Cochrane Information Retrieval Methods Group, the HTA-I Information Specialists’ Group and historic association with such centres as the UK Cochrane Centre, the NHS Centre for Reviews and Dissemination, the Centre for Evidence Based Medicine and the National Institute for Clinical Excellence (NICE). Coupled with the linguistic dominance of English within medical and health science and the science of systematic reviews more generally, this offers a justification for a purposive sample that favours UK, European and Australian guidance documents.

Nine guidance documents were identified. These documents provide guidance for different types of reviews, namely: reviews of interventions, reviews of health technologies, reviews of qualitative research studies, reviews of social science topics, and reviews to inform guidance.

Whilst these guidance documents occasionally offer additional guidance on other types of systematic reviews, we have focused on the core and stated aims of these documents as they relate to literature searching. Table  1 sets out: the guidance document, the version audited, their core stated focus, and a bibliographical pointer to the main guidance relating to literature searching.

Once a list of key guidance documents was determined, it was checked by six senior information professionals based in the UK for relevance to current literature searching in systematic reviews.

Identifying supporting studies

In addition to identifying guidance, the authors sought to populate an evidence base of supporting studies (henceforth referred to as “studies”) that contribute to existing search practice. Studies were first identified by the authors from their knowledge on this topic area and, subsequently, through systematic citation chasing key studies (‘pearls’ [ 1 ]) located within each key stage of the search process. These studies are identified in Additional file  1 : Appendix Table 1. Citation chasing was conducted by analysing the bibliography of references for each study (backwards citation chasing) and through Google Scholar (forward citation chasing). A search of PubMed using the systematic review methods filter was undertaken in August 2017 (see Additional file 1 ). The search terms used were: (literature search*[Title/Abstract]) AND sysrev_methods[sb] and 586 results were returned. These results were sifted for relevance to the key stages in Fig.  1 by CC.

The key stages of literature search guidance as identified from nine key texts

Extracting the data

To reveal the implicit process of literature searching within each guidance document, the relevant sections (chapters) on literature searching were read and re-read, with the aim of determining key methodological stages. We defined a key methodological stage as a distinct step in the overall process for which specific guidance is reported, and action is taken, that collectively would result in a completed literature search.

The chapter or section sub-heading for each methodological stage was extracted into a table using the exact language as reported in each guidance document. The lead author (CC) then read and re-read these data, and the paragraphs of the document to which the headings referred, summarising section details. This table was then reviewed, using comparison and contrast to identify agreements and areas of unique guidance. Consensus across multiple guidelines was used to inform selection of ‘key stages’ in the process of literature searching.

Having determined the key stages to literature searching, we then read and re-read the sections relating to literature searching again, extracting specific detail relating to the methodological process of literature searching within each key stage. Again, the guidance was then read and re-read, first on a document-by-document-basis and, secondly, across all the documents above, to identify both commonalities and areas of unique guidance.

Results and discussion

Our findings.

We were able to identify consensus across the guidance on literature searching for systematic reviews suggesting a shared implicit model within the information retrieval community. Whilst the structure of the guidance varies between documents, the same key stages are reported, even where the core focus of each document is different. We were able to identify specific areas of unique guidance, where a document reported guidance not summarised in other documents, together with areas of consensus across guidance.

Unique guidance

Only one document provided guidance on the topic of when to stop searching [ 2 ]. This guidance from 2005 anticipates a topic of increasing importance with the current interest in time-limited (i.e. “rapid”) reviews. Quality assurance (or peer review) of literature searches was only covered in two guidance documents [ 3 , 4 ]. This topic has emerged as increasingly important as indicated by the development of the PRESS instrument [ 5 ]. Text mining was discussed in four guidance documents [ 4 , 6 , 7 , 8 ] where the automation of some manual review work may offer efficiencies in literature searching [ 8 ].

Agreement between guidance: Defining the key stages of literature searching

Where there was agreement on the process, we determined that this constituted a key stage in the process of literature searching to inform systematic reviews.

From the guidance, we determined eight key stages that relate specifically to literature searching in systematic reviews. These are summarised at Fig. 1 . The data extraction table to inform Fig. 1 is reported in Table  2 . Table 2 reports the areas of common agreement and it demonstrates that the language used to describe key stages and processes varies significantly between guidance documents.

For each key stage, we set out the specific guidance, followed by discussion on how this guidance is situated within the wider literature.

Key stage one: Deciding who should undertake the literature search

The guidance.

Eight documents provided guidance on who should undertake literature searching in systematic reviews [ 2 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ]. The guidance affirms that people with relevant expertise of literature searching should ‘ideally’ be included within the review team [ 6 ]. Information specialists (or information scientists), librarians or trial search co-ordinators (TSCs) are indicated as appropriate researchers in six guidance documents [ 2 , 7 , 8 , 9 , 10 , 11 ].

How the guidance corresponds to the published studies

The guidance is consistent with studies that call for the involvement of information specialists and librarians in systematic reviews [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] and which demonstrate how their training as ‘expert searchers’ and ‘analysers and organisers of data’ can be put to good use [ 13 ] in a variety of roles [ 12 , 16 , 20 , 21 , 24 , 25 , 26 ]. These arguments make sense in the context of the aims and purposes of literature searching in systematic reviews, explored below. The need for ‘thorough’ and ‘replicable’ literature searches was fundamental to the guidance and recurs in key stage two. Studies have found poor reporting, and a lack of replicable literature searches, to be a weakness in systematic reviews [ 17 , 18 , 27 , 28 ] and they argue that involvement of information specialists/ librarians would be associated with better reporting and better quality literature searching. Indeed, Meert et al. [ 29 ] demonstrated that involving a librarian as a co-author to a systematic review correlated with a higher score in the literature searching component of a systematic review [ 29 ]. As ‘new styles’ of rapid and scoping reviews emerge, where decisions on how to search are more iterative and creative, a clear role is made here too [ 30 ].

Knowing where to search for studies was noted as important in the guidance, with no agreement as to the appropriate number of databases to be searched [ 2 , 6 ]. Database (and resource selection more broadly) is acknowledged as a relevant key skill of information specialists and librarians [ 12 , 15 , 16 , 31 ].

Whilst arguments for including information specialists and librarians in the process of systematic review might be considered self-evident, Koffel and Rethlefsen [ 31 ] have questioned if the necessary involvement is actually happening [ 31 ].

Key stage two: Determining the aim and purpose of a literature search

The aim: Five of the nine guidance documents use adjectives such as ‘thorough’, ‘comprehensive’, ‘transparent’ and ‘reproducible’ to define the aim of literature searching [ 6 , 7 , 8 , 9 , 10 ]. Analogous phrases were present in a further three guidance documents, namely: ‘to identify the best available evidence’ [ 4 ] or ‘the aim of the literature search is not to retrieve everything. It is to retrieve everything of relevance’ [ 2 ] or ‘A systematic literature search aims to identify all publications relevant to the particular research question’ [ 3 ]. The Joanna Briggs Institute reviewers’ manual was the only guidance document where a clear statement on the aim of literature searching could not be identified. The purpose of literature searching was defined in three guidance documents, namely to minimise bias in the resultant review [ 6 , 8 , 10 ]. Accordingly, eight of nine documents clearly asserted that thorough and comprehensive literature searches are required as a potential mechanism for minimising bias.

The need for thorough and comprehensive literature searches appears as uniform within the eight guidance documents that describe approaches to literature searching in systematic reviews of effectiveness. Reviews of effectiveness (of intervention or cost), accuracy and prognosis, require thorough and comprehensive literature searches to transparently produce a reliable estimate of intervention effect. The belief that all relevant studies have been ‘comprehensively’ identified, and that this process has been ‘transparently’ reported, increases confidence in the estimate of effect and the conclusions that can be drawn [ 32 ]. The supporting literature exploring the need for comprehensive literature searches focuses almost exclusively on reviews of intervention effectiveness and meta-analysis. Different ‘styles’ of review may have different standards however; the alternative, offered by purposive sampling, has been suggested in the specific context of qualitative evidence syntheses [ 33 ].

What is a comprehensive literature search?

Whilst the guidance calls for thorough and comprehensive literature searches, it lacks clarity on what constitutes a thorough and comprehensive literature search, beyond the implication that all of the literature search methods in Table 2 should be used to identify studies. Egger et al. [ 34 ], in an empirical study evaluating the importance of comprehensive literature searches for trials in systematic reviews, defined a comprehensive search for trials as:

a search not restricted to English language;

where Cochrane CENTRAL or at least two other electronic databases had been searched (such as MEDLINE or EMBASE); and

at least one of the following search methods has been used to identify unpublished trials: searches for (I) conference abstracts, (ii) theses, (iii) trials registers; and (iv) contacts with experts in the field [ 34 ].

Tricco et al. (2008) used a similar threshold of bibliographic database searching AND a supplementary search method in a review when examining the risk of bias in systematic reviews. Their criteria were: one database (limited using the Cochrane Highly Sensitive Search Strategy (HSSS)) and handsearching [ 35 ].

Together with the guidance, this would suggest that comprehensive literature searching requires the use of BOTH bibliographic database searching AND supplementary search methods.

Comprehensiveness in literature searching, in the sense of how much searching should be undertaken, remains unclear. Egger et al. recommend that ‘investigators should consider the type of literature search and degree of comprehension that is appropriate for the review in question, taking into account budget and time constraints’ [ 34 ]. This view tallies with the Cochrane Handbook, which stipulates clearly, that study identification should be undertaken ‘within resource limits’ [ 9 ]. This would suggest that the limitations to comprehension are recognised but it raises questions on how this is decided and reported [ 36 ].

What is the point of comprehensive literature searching?

The purpose of thorough and comprehensive literature searches is to avoid missing key studies and to minimize bias [ 6 , 8 , 10 , 34 , 37 , 38 , 39 ] since a systematic review based only on published (or easily accessible) studies may have an exaggerated effect size [ 35 ]. Felson (1992) sets out potential biases that could affect the estimate of effect in a meta-analysis [ 40 ] and Tricco et al. summarize the evidence concerning bias and confounding in systematic reviews [ 35 ]. Egger et al. point to non-publication of studies, publication bias, language bias and MEDLINE bias, as key biases [ 34 , 35 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ]. Comprehensive searches are not the sole factor to mitigate these biases but their contribution is thought to be significant [ 2 , 32 , 34 ]. Fehrmann (2011) suggests that ‘the search process being described in detail’ and that, where standard comprehensive search techniques have been applied, increases confidence in the search results [ 32 ].

Does comprehensive literature searching work?

Egger et al., and other study authors, have demonstrated a change in the estimate of intervention effectiveness where relevant studies were excluded from meta-analysis [ 34 , 47 ]. This would suggest that missing studies in literature searching alters the reliability of effectiveness estimates. This is an argument for comprehensive literature searching. Conversely, Egger et al. found that ‘comprehensive’ searches still missed studies and that comprehensive searches could, in fact, introduce bias into a review rather than preventing it, through the identification of low quality studies then being included in the meta-analysis [ 34 ]. Studies query if identifying and including low quality or grey literature studies changes the estimate of effect [ 43 , 48 ] and question if time is better invested updating systematic reviews rather than searching for unpublished studies [ 49 ], or mapping studies for review as opposed to aiming for high sensitivity in literature searching [ 50 ].

Aim and purpose beyond reviews of effectiveness

The need for comprehensive literature searches is less certain in reviews of qualitative studies, and for reviews where a comprehensive identification of studies is difficult to achieve (for example, in Public health) [ 33 , 51 , 52 , 53 , 54 , 55 ]. Literature searching for qualitative studies, and in public health topics, typically generates a greater number of studies to sift than in reviews of effectiveness [ 39 ] and demonstrating the ‘value’ of studies identified or missed is harder [ 56 ], since the study data do not typically support meta-analysis. Nussbaumer-Streit et al. (2016) have registered a review protocol to assess whether abbreviated literature searches (as opposed to comprehensive literature searches) has an impact on conclusions across multiple bodies of evidence, not only on effect estimates [ 57 ] which may develop this understanding. It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [ 58 ], and that comprehensive literature searches are not necessarily a marker of literature search quality, as previously thought [ 36 ]. Different approaches to literature searching [ 37 , 38 , 59 , 60 , 61 , 62 ] and developing the concept of when to stop searching are important areas for further study [ 36 , 59 ].

The study by Nussbaumer-Streit et al. has been published since the submission of this literature review [ 63 ]. Nussbaumer-Streit et al. (2018) conclude that abbreviated literature searches are viable options for rapid evidence syntheses, if decision-makers are willing to trade the certainty from a comprehensive literature search and systematic review, but that decision-making which demands detailed scrutiny should still be based on comprehensive literature searches [ 63 ].

Key stage three: Preparing for the literature search

Six documents provided guidance on preparing for a literature search [ 2 , 3 , 6 , 7 , 9 , 10 ]. The Cochrane Handbook clearly stated that Cochrane authors (i.e. researchers) should seek advice from a trial search co-ordinator (i.e. a person with specific skills in literature searching) ‘before’ starting a literature search [ 9 ].

Two key tasks were perceptible in preparing for a literature searching [ 2 , 6 , 7 , 10 , 11 ]. First, to determine if there are any existing or on-going reviews, or if a new review is justified [ 6 , 11 ]; and, secondly, to develop an initial literature search strategy to estimate the volume of relevant literature (and quality of a small sample of relevant studies [ 10 ]) and indicate the resources required for literature searching and the review of the studies that follows [ 7 , 10 ].

Three documents summarised guidance on where to search to determine if a new review was justified [ 2 , 6 , 11 ]. These focused on searching databases of systematic reviews (The Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE)), institutional registries (including PROSPERO), and MEDLINE [ 6 , 11 ]. It is worth noting, however, that as of 2015, DARE (and NHS EEDs) are no longer being updated and so the relevance of this (these) resource(s) will diminish over-time [ 64 ]. One guidance document, ‘Systematic reviews in the Social Sciences’, noted, however, that databases are not the only source of information and unpublished reports, conference proceeding and grey literature may also be required, depending on the nature of the review question [ 2 ].

Two documents reported clearly that this preparation (or ‘scoping’) exercise should be undertaken before the actual search strategy is developed [ 7 , 10 ]).

The guidance offers the best available source on preparing the literature search with the published studies not typically reporting how their scoping informed the development of their search strategies nor how their search approaches were developed. Text mining has been proposed as a technique to develop search strategies in the scoping stages of a review although this work is still exploratory [ 65 ]. ‘Clustering documents’ and word frequency analysis have also been tested to identify search terms and studies for review [ 66 , 67 ]. Preparing for literature searches and scoping constitutes an area for future research.

Key stage four: Designing the search strategy

The Population, Intervention, Comparator, Outcome (PICO) structure was the commonly reported structure promoted to design a literature search strategy. Five documents suggested that the eligibility criteria or review question will determine which concepts of PICO will be populated to develop the search strategy [ 1 , 4 , 7 , 8 , 9 ]. The NICE handbook promoted multiple structures, namely PICO, SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and multi-stranded approaches [ 4 ].

With the exclusion of The Joanna Briggs Institute reviewers’ manual, the guidance offered detail on selecting key search terms, synonyms, Boolean language, selecting database indexing terms and combining search terms. The CEE handbook suggested that ‘search terms may be compiled with the help of the commissioning organisation and stakeholders’ [ 10 ].

The use of limits, such as language or date limits, were discussed in all documents [ 2 , 3 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ].

Search strategy structure

The guidance typically relates to reviews of intervention effectiveness so PICO – with its focus on intervention and comparator - is the dominant model used to structure literature search strategies [ 68 ]. PICOs – where the S denotes study design - is also commonly used in effectiveness reviews [ 6 , 68 ]. As the NICE handbook notes, alternative models to structure literature search strategies have been developed and tested. Booth provides an overview on formulating questions for evidence based practice [ 69 ] and has developed a number of alternatives to the PICO structure, namely: BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory [ 55 ]; SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) for identification of social science and evaluation studies [ 69 ] and, working with Cooke and colleagues, SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) [ 70 ]. SPIDER has been compared to PICO and PICOs in a study by Methley et al. [ 68 ].

The NICE handbook also suggests the use of multi-stranded approaches to developing literature search strategies [ 4 ]. Glanville developed this idea in a study by Whitting et al. [ 71 ] and a worked example of this approach is included in the development of a search filter by Cooper et al. [ 72 ].

Writing search strategies: Conceptual and objective approaches

Hausner et al. [ 73 ] provide guidance on writing literature search strategies, delineating between conceptually and objectively derived approaches. The conceptual approach, advocated by and explained in the guidance documents, relies on the expertise of the literature searcher to identify key search terms and then develop key terms to include synonyms and controlled syntax. Hausner and colleagues set out the objective approach [ 73 ] and describe what may be done to validate it [ 74 ].

The use of limits

The guidance documents offer direction on the use of limits within a literature search. Limits can be used to focus literature searching to specific study designs or by other markers (such as by date) which limits the number of studies returned by a literature search. The use of limits should be described and the implications explored [ 34 ] since limiting literature searching can introduce bias (explored above). Craven et al. have suggested the use of a supporting narrative to explain decisions made in the process of developing literature searches and this advice would usefully capture decisions on the use of search limits [ 75 ].

Key stage five: Determining the process of literature searching and deciding where to search (bibliographic database searching)

Table 2 summarises the process of literature searching as reported in each guidance document. Searching bibliographic databases was consistently reported as the ‘first step’ to literature searching in all nine guidance documents.

Three documents reported specific guidance on where to search, in each case specific to the type of review their guidance informed, and as a minimum requirement [ 4 , 9 , 11 ]. Seven of the key guidance documents suggest that the selection of bibliographic databases depends on the topic of review [ 2 , 3 , 4 , 6 , 7 , 8 , 10 ], with two documents noting the absence of an agreed standard on what constitutes an acceptable number of databases searched [ 2 , 6 ].

The guidance documents summarise ‘how to’ search bibliographic databases in detail and this guidance is further contextualised above in terms of developing the search strategy. The documents provide guidance of selecting bibliographic databases, in some cases stating acceptable minima (i.e. The Cochrane Handbook states Cochrane CENTRAL, MEDLINE and EMBASE), and in other cases simply listing bibliographic database available to search. Studies have explored the value in searching specific bibliographic databases, with Wright et al. (2015) noting the contribution of CINAHL in identifying qualitative studies [ 76 ], Beckles et al. (2013) questioning the contribution of CINAHL to identifying clinical studies for guideline development [ 77 ], and Cooper et al. (2015) exploring the role of UK-focused bibliographic databases to identify UK-relevant studies [ 78 ]. The host of the database (e.g. OVID or ProQuest) has been shown to alter the search returns offered. Younger and Boddy [ 79 ] report differing search returns from the same database (AMED) but where the ‘host’ was different [ 79 ].

The average number of bibliographic database searched in systematic reviews has risen in the period 1994–2014 (from 1 to 4) [ 80 ] but there remains (as attested to by the guidance) no consensus on what constitutes an acceptable number of databases searched [ 48 ]. This is perhaps because thinking about the number of databases searched is the wrong question, researchers should be focused on which databases were searched and why, and which databases were not searched and why. The discussion should re-orientate to the differential value of sources but researchers need to think about how to report this in studies to allow findings to be generalised. Bethel (2017) has proposed ‘search summaries’, completed by the literature searcher, to record where included studies were identified, whether from database (and which databases specifically) or supplementary search methods [ 81 ]. Search summaries document both yield and accuracy of searches, which could prospectively inform resource use and decisions to search or not to search specific databases in topic areas. The prospective use of such data presupposes, however, that past searches are a potential predictor of future search performance (i.e. that each topic is to be considered representative and not unique). In offering a body of practice, this data would be of greater practicable use than current studies which are considered as little more than individual case studies [ 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 ].

When to database search is another question posed in the literature. Beyer et al. [ 91 ] report that databases can be prioritised for literature searching which, whilst not addressing the question of which databases to search, may at least bring clarity as to which databases to search first [ 91 ]. Paradoxically, this links to studies that suggest PubMed should be searched in addition to MEDLINE (OVID interface) since this improves the currency of systematic reviews [ 92 , 93 ]. Cooper et al. (2017) have tested the idea of database searching not as a primary search method (as suggested in the guidance) but as a supplementary search method in order to manage the volume of studies identified for an environmental effectiveness systematic review. Their case study compared the effectiveness of database searching versus a protocol using supplementary search methods and found that the latter identified more relevant studies for review than searching bibliographic databases [ 94 ].

Key stage six: Determining the process of literature searching and deciding where to search (supplementary search methods)

Table 2 also summaries the process of literature searching which follows bibliographic database searching. As Table 2 sets out, guidance that supplementary literature search methods should be used in systematic reviews recurs across documents, but the order in which these methods are used, and the extent to which they are used, varies. We noted inconsistency in the labelling of supplementary search methods between guidance documents.

Rather than focus on the guidance on how to use the methods (which has been summarised in a recent review [ 95 ]), we focus on the aim or purpose of supplementary search methods.

The Cochrane Handbook reported that ‘efforts’ to identify unpublished studies should be made [ 9 ]. Four guidance documents [ 2 , 3 , 6 , 9 ] acknowledged that searching beyond bibliographic databases was necessary since ‘databases are not the only source of literature’ [ 2 ]. Only one document reported any guidance on determining when to use supplementary methods. The IQWiG handbook reported that the use of handsearching (in their example) could be determined on a ‘case-by-case basis’ which implies that the use of these methods is optional rather than mandatory. This is in contrast to the guidance (above) on bibliographic database searching.

The issue for supplementary search methods is similar in many ways to the issue of searching bibliographic databases: demonstrating value. The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged [ 37 , 61 , 62 , 96 , 97 , 98 , 99 , 100 , 101 ] but understanding the value of the search methods to identify studies and data is unclear. In a recently published review, Cooper et al. (2017) reviewed the literature on supplementary search methods looking to determine the advantages, disadvantages and resource implications of using supplementary search methods [ 95 ]. This review also summarises the key guidance and empirical studies and seeks to address the question on when to use these search methods and when not to [ 95 ]. The guidance is limited in this regard and, as Table 2 demonstrates, offers conflicting advice on the order of searching, and the extent to which these search methods should be used in systematic reviews.

Key stage seven: Managing the references

Five of the documents provided guidance on managing references, for example downloading, de-duplicating and managing the output of literature searches [ 2 , 4 , 6 , 8 , 10 ]. This guidance typically itemised available bibliographic management tools rather than offering guidance on how to use them specifically [ 2 , 4 , 6 , 8 ]. The CEE handbook provided guidance on importing data where no direct export option is available (e.g. web-searching) [ 10 ].

The literature on using bibliographic management tools is not large relative to the number of ‘how to’ videos on platforms such as YouTube (see for example [ 102 ]). These YouTube videos confirm the overall lack of ‘how to’ guidance identified in this study and offer useful instruction on managing references. Bramer et al. set out methods for de-duplicating data and reviewing references in Endnote [ 103 , 104 ] and Gall tests the direct search function within Endnote to access databases such as PubMed, finding a number of limitations [ 105 ]. Coar et al. and Ahmed et al. consider the role of the free-source tool, Zotero [ 106 , 107 ]. Managing references is a key administrative function in the process of review particularly for documenting searches in PRISMA guidance.

Key stage eight: Documenting the search

The Cochrane Handbook was the only guidance document to recommend a specific reporting guideline: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 9 ]. Six documents provided guidance on reporting the process of literature searching with specific criteria to report [ 3 , 4 , 6 , 8 , 9 , 10 ]. There was consensus on reporting: the databases searched (and the host searched by), the search strategies used, and any use of limits (e.g. date, language, search filters (The CRD handbook called for these limits to be justified [ 6 ])). Three guidance documents reported that the number of studies identified should be recorded [ 3 , 6 , 10 ]. The number of duplicates identified [ 10 ], the screening decisions [ 3 ], a comprehensive list of grey literature sources searched (and full detail for other supplementary search methods) [ 8 ], and an annotation of search terms tested but not used [ 4 ] were identified as unique items in four documents.

The Cochrane Handbook was the only guidance document to note that the full search strategies for each database should be included in the Additional file 1 of the review [ 9 ].

All guidance documents should ultimately deliver completed systematic reviews that fulfil the requirements of the PRISMA reporting guidelines [ 108 ]. The guidance broadly requires the reporting of data that corresponds with the requirements of the PRISMA statement although documents typically ask for diverse and additional items [ 108 ]. In 2008, Sampson et al. observed a lack of consensus on reporting search methods in systematic reviews [ 109 ] and this remains the case as of 2017, as evidenced in the guidance documents, and in spite of the publication of the PRISMA guidelines in 2009 [ 110 ]. It is unclear why the collective guidance does not more explicitly endorse adherence to the PRISMA guidance.

Reporting of literature searching is a key area in systematic reviews since it sets out clearly what was done and how the conclusions of the review can be believed [ 52 , 109 ]. Despite strong endorsement in the guidance documents, specifically supported in PRISMA guidance, and other related reporting standards too (such as ENTREQ for qualitative evidence synthesis, STROBE for reviews of observational studies), authors still highlight the prevalence of poor standards of literature search reporting [ 31 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 ]. To explore issues experienced by authors in reporting literature searches, and look at uptake of PRISMA, Radar et al. [ 120 ] surveyed over 260 review authors to determine common problems and their work summaries the practical aspects of reporting literature searching [ 120 ]. Atkinson et al. [ 121 ] have also analysed reporting standards for literature searching, summarising recommendations and gaps for reporting search strategies [ 121 ].

One area that is less well covered by the guidance, but nevertheless appears in this literature, is the quality appraisal or peer review of literature search strategies. The PRESS checklist is the most prominent and it aims to develop evidence-based guidelines to peer review of electronic search strategies [ 5 , 122 , 123 ]. A corresponding guideline for documentation of supplementary search methods does not yet exist although this idea is currently being explored.

How the reporting of the literature searching process corresponds to critical appraisal tools is an area for further research. In the survey undertaken by Radar et al. (2014), 86% of survey respondents (153/178) identified a need for further guidance on what aspects of the literature search process to report [ 120 ]. The PRISMA statement offers a brief summary of what to report but little practical guidance on how to report it [ 108 ]. Critical appraisal tools for systematic reviews, such as AMSTAR 2 (Shea et al. [ 124 ]) and ROBIS (Whiting et al. [ 125 ]), can usefully be read alongside PRISMA guidance, since they offer greater detail on how the reporting of the literature search will be appraised and, therefore, they offer a proxy on what to report [ 124 , 125 ]. Further research in the form of a study which undertakes a comparison between PRISMA and quality appraisal checklists for systematic reviews would seem to begin addressing the call, identified by Radar et al., for further guidance on what to report [ 120 ].

Limitations

Other handbooks exist.

A potential limitation of this literature review is the focus on guidance produced in Europe (the UK specifically) and Australia. We justify the decision for our selection of the nine guidance documents reviewed in this literature review in section “ Identifying guidance ”. In brief, these nine guidance documents were selected as the most relevant health care guidance that inform UK systematic reviewing practice, given that the UK occupies a prominent position in the science of health information retrieval. We acknowledge the existence of other guidance documents, such as those from North America (e.g. the Agency for Healthcare Research and Quality (AHRQ) [ 126 ], The Institute of Medicine [ 127 ] and the guidance and resources produced by the Canadian Agency for Drugs and Technologies in Health (CADTH) [ 128 ]). We comment further on this directly below.

The handbooks are potentially linked to one another

What is not clear is the extent to which the guidance documents inter-relate or provide guidance uniquely. The Cochrane Handbook, first published in 1994, is notably a key source of reference in guidance and systematic reviews beyond Cochrane reviews. It is not clear to what extent broadening the sample of guidance handbooks to include North American handbooks, and guidance handbooks from other relevant countries too, would alter the findings of this literature review or develop further support for the process model. Since we cannot be clear, we raise this as a potential limitation of this literature review. On our initial review of a sample of North American, and other, guidance documents (before selecting the guidance documents considered in this review), however, we do not consider that the inclusion of these further handbooks would alter significantly the findings of this literature review.

This is a literature review

A further limitation of this review was that the review of published studies is not a systematic review of the evidence for each key stage. It is possible that other relevant studies could help contribute to the exploration and development of the key stages identified in this review.

This literature review would appear to demonstrate the existence of a shared model of the literature searching process in systematic reviews. We call this model ‘the conventional approach’, since it appears to be common convention in nine different guidance documents.

The findings reported above reveal eight key stages in the process of literature searching for systematic reviews. These key stages are consistently reported in the nine guidance documents which suggests consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews.

In Table 2 , we demonstrate consensus regarding the application of literature search methods. All guidance documents distinguish between primary and supplementary search methods. Bibliographic database searching is consistently the first method of literature searching referenced in each guidance document. Whilst the guidance uniformly supports the use of supplementary search methods, there is little evidence for a consistent process with diverse guidance across documents. This may reflect differences in the core focus across each document, linked to differences in identifying effectiveness studies or qualitative studies, for instance.

Eight of the nine guidance documents reported on the aims of literature searching. The shared understanding was that literature searching should be thorough and comprehensive in its aim and that this process should be reported transparently so that that it could be reproduced. Whilst only three documents explicitly link this understanding to minimising bias, it is clear that comprehensive literature searching is implicitly linked to ‘not missing relevant studies’ which is approximately the same point.

Defining the key stages in this review helps categorise the scholarship available, and it prioritises areas for development or further study. The supporting studies on preparing for literature searching (key stage three, ‘preparation’) were, for example, comparatively few, and yet this key stage represents a decisive moment in literature searching for systematic reviews. It is where search strategy structure is determined, search terms are chosen or discarded, and the resources to be searched are selected. Information specialists, librarians and researchers, are well placed to develop these and other areas within the key stages we identify.

This review calls for further research to determine the suitability of using the conventional approach. The publication dates of the guidance documents which underpin the conventional approach may raise questions as to whether the process which they each report remains valid for current systematic literature searching. In addition, it may be useful to test whether it is desirable to use the same process model of literature searching for qualitative evidence synthesis as that for reviews of intervention effectiveness, which this literature review demonstrates is presently recommended best practice.

Abbreviations

Behaviour of interest; Health context; Exclusions; Models or Theories

Cochrane Database of Systematic Reviews

The Cochrane Central Register of Controlled Trials

Database of Abstracts of Reviews of Effects

Enhancing transparency in reporting the synthesis of qualitative research

Institute for Quality and Efficiency in Healthcare

National Institute for Clinical Excellence

Population, Intervention, Comparator, Outcome

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Setting, Perspective, Intervention, Comparison, Evaluation

Sample, Phenomenon of Interest, Design, Evaluation, Research type

STrengthening the Reporting of OBservational studies in Epidemiology

Trial Search Co-ordinators

Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Information & Libraries Journal. 2008;25(4):313–7.

Article   Google Scholar  

Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell Publishing Ltd; 2006.

Book   Google Scholar  

Institute for Quality and Efficiency in Health Care (IQWiG). IQWiG Methods Resources. 7 Information retrieval 2014 [Available from: https://www.ncbi.nlm.nih.gov/books/NBK385787/ .

NICE: National Institute for Health and Care Excellence. Developing NICE guidelines: the manual 2014. Available from: https://www.nice.org.uk/media/default/about/what-we-do/our-programmes/developing-nice-guidelines-the-manual.pdf .

Sampson M. MJ, Lefebvre C, Moher D, Grimshaw J. Peer Review of Electronic Search Strategies: PRESS; 2008.

Google Scholar  

Centre for Reviews & Dissemination. Systematic reviews – CRD’s guidance for undertaking reviews in healthcare. York: Centre for Reviews and Dissemination, University of York; 2009.

eunetha: European Network for Health Technology Assesment Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness 2016. Available from: http://www.eunethta.eu/sites/default/files/Guideline_Information_Retrieval_V1-1.pdf .

Kugley SWA, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: Campbell Collaboration. 2017; Available from: https://www.campbellcollaboration.org/library/searching-for-studies-information-retrieval-guide-campbell-reviews.html

Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: JPT H, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions; 2011.

Collaboration for Environmental Evidence. Guidelines for Systematic Review and Evidence Synthesis in Environmental Management.: Environmental Evidence:; 2013. Available from: http://www.environmentalevidence.org/wp-content/uploads/2017/01/Review-guidelines-version-4.2-final-update.pdf .

The Joanna Briggs Institute. Joanna Briggs institute reviewers’ manual. 2014th ed: the Joanna Briggs institute; 2014. Available from: https://joannabriggs.org/assets/docs/sumari/ReviewersManual-2014.pdf

Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Inf Libr J. 2003;20(2):65–74.

Article   CAS   Google Scholar  

Harris MR. The librarian's roles in the systematic review process: a case study. Journal of the Medical Library Association. 2005;93(1):81–7.

PubMed   PubMed Central   Google Scholar  

Egger JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015;10(5):e0125931.

Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7.

Article   PubMed   Google Scholar  

McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74–80.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617–26.

Weller AC. Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports. J Med Libr Assoc. 2004;92(2):163–4.

Swinkels A, Briddon J, Hall J. Two physiotherapists, one librarian and a systematic literature review: collaboration in action. Health Info Libr J. 2006;23(4):248–56.

Foster M. An overview of the role of librarians in systematic reviews: from expert search to project manager. EAHIL. 2015;11(3):3–7.

Lawson L. OPERATING OUTSIDE LIBRARY WALLS 2004.

Vassar M, Yerokhin V, Sinnett PM, Weiher M, Muckelrath H, Carr B, et al. Database selection in systematic reviews: an insight through clinical neurology. Health Inf Libr J. 2017;34(2):156–64.

Townsend WA, Anderson PF, Ginier EC, MacEachern MP, Saylor KM, Shipman BL, et al. A competency framework for librarians involved in systematic reviews. Journal of the Medical Library Association : JMLA. 2017;105(3):268–75.

Cooper ID, Crum JA. New activities and changing roles of health sciences librarians: a systematic review, 1990-2012. Journal of the Medical Library Association : JMLA. 2013;101(4):268–77.

Crum JA, Cooper ID. Emerging roles for biomedical librarians: a survey of current practice, challenges, and changes. Journal of the Medical Library Association : JMLA. 2013;101(4):278–86.

Dudden RF, Protzko SL. The systematic review team: contributions of the health sciences librarian. Med Ref Serv Q. 2011;30(3):301–15.

Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440–8.

Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Academic medicine : journal of the Association of American Medical Colleges. 2011;86(8):1049–54.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association : JMLA. 2016;104(4):267–77.

Morris M, Boruff JT, Gore GC. Scoping reviews: establishing the role of the librarian. Journal of the Medical Library Association : JMLA. 2016;104(4):346–54.

Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016;11(9):e0163309.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Fehrmann P, Thomas J. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods. 2011;2(1):15–32.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews. 2016;5(1):74.

Article   PubMed   PubMed Central   Google Scholar  

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment (Winchester, England). 2003;7(1):1–76.

Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. J Clin Epidemiol. 2008;61(5):422–34.

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care. 2010;26(4):431–5.

Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Inf Libr J. 2010;27(2):114–22.

Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Systematic Reviews. 2015;4(1):36.

Betrán AP, Say L, Gülmezoglu AM, Allen T, Hampson L. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005;5

Felson DT. Bias in meta-analytic research. J Clin Epidemiol. 1992;45(8):885–92.

Article   PubMed   CAS   Google Scholar  

Franco A, Malhotra N, Simonovits G. Publication bias in the social sciences: unlocking the file drawer. Science. 2014;345(6203):1502–5.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.

Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, et al. Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One. 2017;12(4):e0176210.

Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet (London, England). 1997;350(9074):326–9.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health technology assessment (Winchester, England). 2003;7(41):1–90.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76.

Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki A-A, Ioannidis JPA. The effects of excluding treatments from network meta-analyses: survey. BMJ : British Medical Journal. 2013;347

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127.

van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(8):838–44.e3.

Buchberger B, Krabbe L, Lux B, Mattivi JT. Evidence mapping for decision making: feasibility versus accuracy - when to abandon high sensitivity in electronic searches. German medical science : GMS e-journal. 2016;14:Doc09.

Lorenc T, Pearson M, Jamal F, Cooper C, Garside R. The role of systematic reviews of qualitative evidence in evaluating interventions: a case study. Research Synthesis Methods. 2012;3(1):1–10.

Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Res Pap Educ. 2007;22(2):213–28.

Barroso J, Gollop CJ, Sandelowski M, Meynell J, Pearce PF, Collins LJ. The challenges of searching for and retrieving qualitative studies. West J Nurs Res. 2003;25(2):153–78.

Britten N, Garside R, Pope C, Frost J, Cooper C. Asking more of qualitative synthesis: a response to Sally Thorne. Qual Health Res. 2017;27(9):1370–6.

Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015;32(3):220–35.

Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014;3:135.

Nussbaumer-Streit B, Klerings I, Wagner G, Titscher V, Gartlehner G. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Systematic Reviews. 2016;5:197.

Wagner G, Nussbaumer-Streit B, Greimel J, Ciapponi A, Gartlehner G. Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017;17(1):121.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19(4):591–603.

Pearson M, Moxham T, Ashton K. Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Eval Health Prof. 2011;34(3):297–308.

Levay P, Raynor M, Tuvey D. The Contributions of MEDLINE, Other Bibliographic Databases and Various Search Techniques to NICE Public Health Guidance. 2015. 2015;10(1):19.

Nussbaumer-Streit B, Klerings I, Wagner G, Heise TL, Dobrescu AI, Armijo-Olivo S, et al. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol. 2018;102:1–11.

Briscoe S, Cooper C, Glanville J, Lefebvre C. The loss of the NHS EED and DARE databases and the effect on evidence synthesis and evaluation. Res Synth Methods. 2017;8(3):256–7.

Stansfield C, O'Mara-Eves A, Thomas J. Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods.n/a-n/a.

Petrova M, Sutcliffe P, Fulford KW, Dale J. Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: a word frequency analysis study. Journal of the American Medical Informatics Association : JAMIA. 2012;19(3):479–88.

Stansfield C, Thomas J, Kavanagh J. 'Clustering' documents automatically to support scoping reviews of research: a case study. Res Synth Methods. 2013;4(3):230–41.

PubMed   Google Scholar  

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Andrew B. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech. 2006;24(3):355–68.

Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22(10):1435–43.

Whiting P, Westwood M, Bojke L, Palmer S, Richardson G, Cooper J, et al. Clinical effectiveness and cost-effectiveness of tests for the diagnosis and investigation of urinary tract infection in children: a systematic review and economic model. Health technology assessment (Winchester, England). 2006;10(36):iii-iv, xi-xiii, 1–154.

Cooper C, Levay P, Lorenc T, Craig GM. A population search filter for hard-to-reach populations increased search efficiency for a systematic review. J Clin Epidemiol. 2014;67(5):554–9.

Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Systematic Reviews. 2012;1(1):19.

Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one. J Clin Epidemiol. 2016;77:118–24.

Craven J, Levay P. Recording database searches for systematic reviews - what is the value of adding a narrative to peer-review checklists? A case study of nice interventional procedures guidance. Evid Based Libr Inf Pract. 2011;6(4):72–87.

Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Syst Rev. 2015;4:104.

Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, et al. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013;66(9):1051–7.

Cooper C, Rogers M, Bethel A, Briscoe S, Lowe J. A mapping review of the literature on UK-focused health and social care databases. Health Inf Libr J. 2015;32(1):5–22.

Younger P, Boddy K. When is a search not a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Inf Libr J. 2009;26(2):126–35.

Lam MT, McDiarmid M. Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. Journal of the Medical Library Association : JMLA. 2016;104(4):284–9.

Bethel A, editor Search summary tables for systematic reviews: results and findings. HLC Conference 2017a.

Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16(1):161.

Adams CE, Frederick K. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994;24(3):741–8.

Kelly L, St Pierre-Hansen N. So many databases, such little clarity: searching the literature for the topic aboriginal. Canadian family physician Medecin de famille canadien. 2008;54(11):1572–3.

Lawrence DW. What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention. 2008;14(6):401–4.

Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56(10):943–55.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine. 2004;12(4):228–32.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476–87.

Taylor B, Wylie E, Dempster M, Donnelly M. Systematically retrieving research: a case study evaluating seven databases. Res Soc Work Pract. 2007;17(6):697–706.

Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Info Libr J. 2013;30(1):49–58.

Duffy S, de Kock S, Misso K, Noake C, Ross J, Stirk L. Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE in-process searches via Ovid. Journal of the Medical Library Association : JMLA. 2016;104(4):309–12.

Katchamart W, Faulkner A, Feldman B, Tomlinson G, Bombardier C. PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews. J Clin Epidemiol. 2011;64(7):805–7.

Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement (in Press). Research Synthesis Methods. 2017;

Cooper C, Booth, A., Britten, N., Garside, R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: A methodological review. (In Press). BMC Systematic Reviews. 2017.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ (Clinical research ed). 2005;331(7524):1064–5.

Article   PubMed Central   Google Scholar  

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. PharmacoEconomics. 2015;33(1):5–11.

Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with web of science and Google scholar. Res Synth Methods. 2016;7(1):34–45.

McManus RJ, Wilson S, Delaney BC, Fitzmaurice DA, Hyde CJ, Tobias RS, et al. Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ (Clinical research ed). 1998;317(7172):1562–3.

Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. Journal of public health research. 2014;3(2):177.

Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, et al. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libr Rev. 1999;16(2):112–20.

Bethel A. Endnote Training (YouTube Videos) 2017b [Available from: http://medicine.exeter.ac.uk/esmi/workstreams/informationscience/is_resources,_guidance_&_advice/ .

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association : JMLA. 2016;104(3):240–3.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. Journal of the Medical Library Association : JMLA. 2017;105(1):84–7.

Gall C, Brahmi FA. Retrieval comparison of EndNote to search MEDLINE (Ovid and PubMed) versus searching them directly. Medical reference services quarterly. 2004;23(3):25–32.

Ahmed KK, Al Dhubaib BE. Zotero: a bibliographic assistant to researcher. J Pharmacol Pharmacother. 2011;2(4):303–5.

Coar JT, Sewell JP. Zotero: harnessing the power of a personal bibliographic manager. Nurse Educ. 2010;35(5):205–7.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748–54.

Toews LC. Compliance of systematic reviews in veterinary journals with preferred reporting items for systematic reviews and meta-analysis (PRISMA) literature search reporting guidelines. Journal of the Medical Library Association : JMLA. 2017;105(3):233–9.

Booth A. "brimful of STARLITE": toward standards for reporting literature searches. Journal of the Medical Library Association : JMLA. 2006;94(4):421–9. e205

Faggion CM Jr, Wu YC, Tu YK, Wasiak J. Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery. Br J Radiol. 2016;89(1062):20150878.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods. 2014;5(2):116–30.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association : JMLA. 2009;97(1):21–9.

Bigna JJ, Um LN, Nansseu JR. A comparison of quality of abstracts of systematic reviews including meta-analysis of randomized controlled trials in high-impact general medicine journals before and after the publication of PRISMA extension for abstracts: a systematic review and meta-analysis. Syst Rev. 2016;5(1):174.

Akhigbe T, Zolnourian A, Bulters D. Compliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: review of literature. Journal of clinical neuroscience : official journal of the Neurosurgical Society of Australasia. 2017;39:45–8.

Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals' instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One. 2011;6(11):e27611.

Wasiak J, Tyack Z, Ware R. Goodwin N. Jr. Poor methodological quality and reporting standards of systematic reviews in burn care management. International wound journal: Faggion CM; 2016.

Tam WW, Lo KK, Khalechelvam P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study. BMJ Open. 2017;7(2):e013905.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Methods. 2014;5(2):98–115.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods. 2015;6(1):87–95.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358.

Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

Relevo R, Balshem H. Finding evidence for comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol. 2011;64(11):1168–77.

Medicine Io. Standards for Systematic Reviews 2011 [Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx .

CADTH: Resources 2018.

Download references

Acknowledgements

CC acknowledges the supervision offered by Professor Chris Hyde.

This publication forms a part of CC’s PhD. CC’s PhD was funded through the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Project Number 16/54/11). The open access fee for this publication was paid for by Exeter Medical School.

RG and NB were partially supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula.

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Author information

Authors and affiliations.

Institute of Health Research, University of Exeter Medical School, Exeter, UK

Chris Cooper & Jo Varley-Campbell

HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

Andrew Booth

Nicky Britten

European Centre for Environment and Human Health, University of Exeter Medical School, Truro, UK

Ruth Garside

You can also search for this author in PubMed   Google Scholar

Contributions

CC conceived the idea for this study and wrote the first draft of the manuscript. CC discussed this publication in PhD supervision with AB and separately with JVC. CC revised the publication with input and comments from AB, JVC, RG and NB. All authors revised the manuscript prior to submission. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chris Cooper .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Appendix tables and PubMed search strategy. Key studies used for pearl growing per key stage, working data extraction tables and the PubMed search strategy. (DOCX 30 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cooper, C., Booth, A., Varley-Campbell, J. et al. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 18 , 85 (2018). https://doi.org/10.1186/s12874-018-0545-3

Download citation

Received : 20 September 2017

Accepted : 06 August 2018

Published : 14 August 2018

DOI : https://doi.org/10.1186/s12874-018-0545-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Literature Search Process
• Citation Chasing
• Tacit Models
• Unique Guidance
• Information Specialists

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

(Stanford users can avoid this Captcha by logging in.)

• Send to text email RefWorks EndNote printer

Systematic approaches to a successful literature review

Available online, at the library.

Education Library (Cubberley)

The Education Library is closed for construction. Request items for pickup at another library.

More options

• Find it at other libraries via WorldCat
• Contributors

Description

Creators/contributors, contents/summary.

• 1. The literature review : its role within research
• 2. Systematic approaches to the literature
• 3. Planning and writing a literature review
• 4. Defining the scope
• 5. Searching the literature
• 6. Assessing the evidence base
• 7. Synthesising included studies
• 8. Analysing the findings
• 9. Writing up and presenting data
• 10. Managing the literature project.

Bibliographic information

Browse related items.

• Stanford Home
• Maps & Directions
• Search Stanford
• Emergency Info
• Terms of Use
• Non-Discrimination
• Accessibility

© Stanford University , Stanford , California 94305 .

• Request new password
• Create a new account

Systematic Approaches to a Successful Literature Review

Student resources.

The resources on this website have been specifically designed to support your study:

• Answers to exercises
• Review Protocol Template

Disclaimer:

This website may contain links to both internal and external websites. All links included were active at the time the website was launched. SAGE does not operate these external websites and does not necessarily endorse the views expressed within them. SAGE cannot take responsibility for the changing content or nature of linked sites, as these sites are outside of our control and subject to change without our knowledge. If you do find an inactive link to an external website, please try to locate that website by using a search engine. SAGE will endeavour to update inactive or broken links when possible.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Car-t cells therapy in glioblastoma: a systematic review on molecular targets and treatment strategies.

1. Introduction

2. materials and methods, 2.1. literature review, 2.2. data extraction, 2.3. outcomes, 2.4. risk of bias assessment, 2.5. statistical analysis, 3.1. literature review, 3.2. data analysis, 4. discussion, 4.1. molecular mechanisms of car-t cell immunotherapy, 4.2. targeted therapies proposed using car-t cells, 4.3. limitations and future perspectives, 5. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

Click here to enlarge figure

• Yuan, B.; Wang, G.; Tang, X.; Tong, A.; Zhou, L. Immunotherapy of Glioblastoma: Recent Advances and Future Prospects. Hum. Vaccines Immunother. 2022 , 18 , 2055417. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Vik-Mo, E.O.; Nyakas, M.; Mikkelsen, B.V.; Moe, M.C.; Due-Tønnesen, P.; Suso, E.M.I.; Sæbøe-Larssen, S.; Sandberg, C.; Brinchmann, J.E.; Helseth, E.; et al. Therapeutic Vaccination against Autologous Cancer Stem Cells with mRNA-Transfected Dendritic Cells in Patients with Glioblastoma. Cancer Immunol. Immunother. 2013 , 62 , 1499–1509. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Markovic, D.S.; Glass, R.; Synowitz, M.; van Rooijen, N.; Kettenmann, H. Microglia Stimulate the Invasiveness of Glioma Cells by Increasing the Activity of Metalloprotease-2. J. Neuropathol. Exp. Neurol. 2005 , 64 , 754–762. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Luksik, A.S.; Yazigi, E.; Shah, P.; Jackson, C.M. CAR T Cell Therapy in Glioblastoma: Overcoming Challenges Related to Antigen Expression. Cancers 2023 , 15 , 1414. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews. BMJ 2021 , 372 , n71. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Stang, A. Critical Evaluation of the Newcastle-Ottawa Scale for the Assessment of the Quality of Nonrandomized Studies in Meta-Analyses. Eur. J. Epidemiol. 2010 , 25 , 603–605. [ Google Scholar ] [ CrossRef ]
• Ahmed, N.; Brawley, V.; Hegde, M.; Bielamowicz, K.; Kalra, M.; Landi, D.; Robertson, C.; Gray, T.L.; Diouf, O.; Wakefield, A.; et al. HER2-Specific Chimeric Antigen Receptor—Modified Virus-Specific T Cells for Progressive Glioblastoma: A Phase 1 Dose-Escalation Trial. JAMA Oncol. 2017 , 3 , 1094–1101. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Brown, C.E.; Badie, B.; Barish, M.E.; Weng, L.; Ostberg, J.R.; Chang, W.-C.; Naranjo, A.; Starr, R.; Wagner, J.; Wright, C.; et al. Bioactivity and Safety of IL13Rα2-Redirected Chimeric Antigen Receptor CD8+ T Cells in Patients with Recurrent Glioblastoma. Clin. Cancer Res. 2015 , 21 , 4062–4072. [ Google Scholar ] [ CrossRef ]
• Brown, C.E.; Alizadeh, D.; Starr, R.; Weng, L.; Wagner, J.R.; Naranjo, A.; Ostberg, J.R.; Blanchard, M.S.; Kilpatrick, J.; Simpson, J.; et al. Regression of Glioblastoma after Chimeric Antigen Receptor T-Cell Therapy. N. Engl. J. Med. 2016 , 375 , 2561–2569. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• O’Rourke, D.M.; Nasrallah, M.P.; Desai, A.; Melenhorst, J.J.; Mansfield, K.; Morrissette, J.J.D.; Martinez-Lage, M.; Brem, S.; Maloney, E.; Shen, A.; et al. A Single Dose of Peripherally Infused EGFRvIII-Directed CAR-T cells Mediates Antigen Loss and Induces Adaptive Resistance in Patients with Recurrent Glioblastoma. Sci. Transl. Med. 2017 , 9 , eaaa0984. [ Google Scholar ] [ CrossRef ]
• Lin, Q.; Ba, T.; Ho, J.; Chen, D.; Cheng, Y.; Wang, L.; Xu, G.; Xu, L.; Zhou, Y.; Wei, Y.; et al. First-in-Human Trial of EphA2-Redirected CAR T-Cells in Patients with Recurrent Glioblastoma: A Preliminary Report of Three Cases at the Starting Dose. Front. Oncol. 2021 , 11 , 694941. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Wang, S.; O’Rourke, D.M.; Chawla, S.; Verma, G.; Nasrallah, M.P.; Morrissette, J.J.D.; Plesa, G.; June, C.H.; Brem, S.; Maloney, E.; et al. Multiparametric Magnetic Resonance Imaging in the Assessment of Anti-EGFRvIII Chimeric Antigen Receptor T Cell Therapy in Patients with Recurrent Glioblastoma. Br. J. Cancer 2019 , 120 , 54–56. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Goff, S.L.; Morgan, R.A.; Yang, J.C.; Sherry, R.M.; Robbins, P.F.; Restifo, N.P.; Feldman, S.A.; Lu, Y.-C.; Lu, L.; Zheng, Z.; et al. Pilot Trial of Adoptive Transfer of Chimeric Antigen Receptor–Transduced T Cells Targeting EGFRvIII in Patients with Glioblastoma. J. Immunother. 2019 , 42 , 126–135. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Liu, Z.; Zhou, J.; Yang, X.; Liu, Y.; Zou, C.; Lv, W.; Chen, C.; Cheng, K.K.; Chen, T.; Chang, L.-J.; et al. Safety and Antitumor Activity of GD2-Specific 4SCAR-T Cells in Patients with Glioblastoma. Mol. Cancer 2023 , 22 , 3. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Agosti, E.; Zeppieri, M.; De Maria, L.; Tedeschi, C.; Fontanella, M.M.; Panciani, P.P.; Ius, T. Glioblastoma Immunotherapy: A Systematic Review of the Present Strategies and Prospects for Advancements. Int. J. Mol. Sci. 2023 , 24 , 15037. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Agosti, E.; Panciani, P.P.; Zeppieri, M.; De Maria, L.; Pasqualetti, F.; Tel, A.; Zanin, L.; Fontanella, M.M.; Ius, T. Tumor Microenvironment and Glioblastoma Cell Interplay as Promoters of Therapeutic Resistance. Biology 2023 , 12 , 736. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• June, C.H.; O’Connor, R.S.; Kawalekar, O.U.; Ghassemi, S.; Milone, M.C. CAR T Cell Immunotherapy for Human Cancer. Science 2018 , 359 , 1361–1365. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Shannon, M.L.; Noelle, F.; Shaw, P.A.; Richard, A.; Barrett, D.M.; Bunin, N.J.; Chew, A.; Gonzalez, V.E.; Zheng, Z.; Lacey, S.F.; et al. Chimeric Antigen Receptor T Cells for Sustained Remissions in Leukemia. N. Engl. J. Med. 2014 , 371 , 1507–1517. [ Google Scholar ] [ CrossRef ]
• Guedan, S.; Posey, A.D.; Shaw, C.; Wing, A.; Da, T.; Patel, P.R.; McGettigan, S.E.; Casado-Medrano, V.; Kawalekar, O.U.; Uribe-Herranz, M.; et al. Enhancing CAR T Cell Persistence through ICOS and 4-1BB Costimulation. JCI Insight 2018 , 3 , e96976. [ Google Scholar ] [ CrossRef ]
• Roybal, K.T.; Rupp, L.J.; Morsut, L.; Walker, W.J.; McNally, K.A.; Park, J.S.; Lim, W.A. Precision Tumor Recognition by T Cells with Combinatorial Antigen-Sensing Circuits. Cell 2016 , 164 , 770–779. [ Google Scholar ] [ CrossRef ]
• Neelapu, S.S.; Tummala, S.; Kebriaei, P.; Wierda, W.; Gutierrez, C.; Locke, F.L.; Komanduri, K.V.; Lin, Y.; Jain, N.; Daver, N.; et al. Chimeric Antigen Receptor T-Cell Therapy—Assessment and Management of Toxicities. Nat. Rev. Clin. Oncol. 2018 , 15 , 47–62. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Guo, X.; Chang, M.; Wang, Y.; Xing, B.; Ma, W. B7-H3 in Brain Malignancies: Immunology and Immunotherapy. Int. J. Biol. Sci. 2023 , 19 , 3762–3780. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Majzner, R.G.; Theruvath, J.L.; Nellan, A.; Heitzeneder, S.; Cui, Y.; Mount, C.W.; Rietberg, S.P.; Linde, M.H.; Xu, P.; Rota, C.; et al. CAR-T cells Targeting B7-H3, A Pan-Cancer Antigen, Demonstrate Potent Preclinical Activity against Pediatric Solid Tumors and Brain Tumors. Clin. Cancer Res. Off. J. Am. Assoc. Cancer Res. 2019 , 25 , 2560–2574. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Du, H.; Hirabayashi, K.; Ahn, S.; Kren, N.P.; Montgomery, S.A.; Wang, X.; Tiruthani, K.; Mirlekar, B.; Michaud, D.; Greene, K.; et al. Antitumor Responses in the Absence of Toxicity in Solid Tumors by Targeting B7-H3 via Chimeric Antigen Receptor T Cells. Cancer Cell 2019 , 35 , 221. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• An, Z.; Aksoy, O.; Zheng, T.; Fan, Q.W.; Weiss, W.A. Epidermal Growth Factor Receptor and EGFRvIII in Glioblastoma: Signaling Pathways and Targeted Therapies. Oncogene 2018 , 37 , 1561–1575. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Chuntova, P.; Hou, Y.; Naka, R.; Yamamichi, A.; Chen, T.; Goretsky, Y.; Hatae, R.; Nejo, T.; Kohanbash, G.; Mende, A.L.; et al. Novel EGFRvIII-CAR Transgenic Mice for Rigorous Preclinical Studies in Syngeneic Mice. Neuro-Oncol. 2022 , 24 , 259–272. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zeng, J.; Zhang, J.; Yang, Y.-Z.; Wang, F.; Jiang, H.; Chen, H.-D.; Wu, H.-Y.; Sai, K.; Hu, W.-M. IL13RA2 Is Overexpressed in Malignant Gliomas and Related to Clinical Outcome of Patients. Am. J. Transl. Res. 2020 , 12 , 4702–4714. [ Google Scholar ] [ PubMed ]
• Kim, K.; Gwak, H.-S.; Han, N.; Hong, E.K.; Choi, B.K.; Lee, S.; Choi, S.; Park, J.-H.; Seok, J.-H.; Jeon, Y.; et al. Chimeric Antigen Receptor T Cells with Modified Interleukin-13 Preferentially Recognize IL13Rα2 and Suppress Malignant Glioma: A Preclinical Study. Front. Immunol. 2021 , 12 , 715000. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Agosti, E.; Zeppieri, M.; Ghidoni, M.; Ius, T.; Tel, A.; Fontanella, M.M.; Panciani, P.P. Role of Glioma Stem Cells in Promoting Tumor Chemo- and Radioresistance: A Systematic Review of Potential Targeted Treatments. World J. Stem Cells 2024 , 16 , 604–614. [ Google Scholar ] [ CrossRef ]
• De Maria, L.; Panciani, P.P.; Zeppieri, M.; Ius, T.; Serioli, S.; Piazza, A.; Di Giovanni, E.; Fontanella, M.M.; Agosti, E. A Systematic Review of the Metabolism of High-Grade Gliomas: Current Targeted Therapies and Future Perspectives. Int. J. Mol. Sci. 2024 , 25 , 724. [ Google Scholar ] [ CrossRef ]

Share and Cite

Agosti, E.; Garaba, A.; Antonietti, S.; Ius, T.; Fontanella, M.M.; Zeppieri, M.; Panciani, P.P. CAR-T Cells Therapy in Glioblastoma: A Systematic Review on Molecular Targets and Treatment Strategies. Int. J. Mol. Sci. 2024 , 25 , 7174. https://doi.org/10.3390/ijms25137174

Agosti E, Garaba A, Antonietti S, Ius T, Fontanella MM, Zeppieri M, Panciani PP. CAR-T Cells Therapy in Glioblastoma: A Systematic Review on Molecular Targets and Treatment Strategies. International Journal of Molecular Sciences . 2024; 25(13):7174. https://doi.org/10.3390/ijms25137174

Agosti, Edoardo, Alexandru Garaba, Sara Antonietti, Tamara Ius, Marco Maria Fontanella, Marco Zeppieri, and Pier Paolo Panciani. 2024. "CAR-T Cells Therapy in Glioblastoma: A Systematic Review on Molecular Targets and Treatment Strategies" International Journal of Molecular Sciences 25, no. 13: 7174. https://doi.org/10.3390/ijms25137174

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.112(1); Jan-Feb 2015

Systematically Reviewing the Literature: Building the Evidence for Health Care Quality

There are important research and non-research reasons to systematically review the literature. This article describes a step-by-step process to systematically review the literature along with links to key resources. An example of a graduate program using systematic literature reviews to link research and quality improvement practices is also provided.

Introduction

Systematic reviews that summarize the available information on a topic are an important part of evidence-based health care. There are both research and non-research reasons for undertaking a literature review. It is important to systematically review the literature when one would like to justify the need for a study, to update personal knowledge and practice, to evaluate current practices, to develop and update guidelines for practice, and to develop work related policies. 1 A systematic review draws upon the best health services research principles and methods to address: What is the state of the evidence on the selected topic? The systematic process enables others to reproduce the methods and to make a rational determination of whether to accept the results of the review. An abundance of articles on systematic reviews exist focusing on different aspects of systematic reviews. 2 – 9 The purpose of this article is to describe a step by step process of systematically reviewing the health care literature and provide links to key resources.

Systematic Review Process: Six Key Steps

Six key steps to systematically review the literature are outlined in Table 1 and discussed here.

Systematic Review Steps

1. Formulate the Question and Refine the Topic

When preparing a topic to conduct a systematic review, it is important to ask at the outset, “What exactly am I looking for?” Hopefully it seems like an obvious step, but explicitly writing a one or two sentence statement of the topic before you begin to search is often overlooked. It is important for several reasons; in particular because, although we usually think we know what we are searching for, in truth our mental image of a topic is often quite fuzzy. The act of writing something concise and intelligible to a reader, even if you are the only one who will read it, clarifies your thoughts and can inspire you to ask key questions. In addition, in subsequent steps of the review process, when you begin to develop a strategy for searching the literature, your topic statement is the ready raw material from which you can extract the key concepts and terminology for your strategies. The medical and related health literature is massive, so the more precise and specific your understanding of your information need, the better your results will be when you search.

2. Search, Retrieve, and Select Relevant Articles

The retrieval tools chosen to search the literature should be determined by the purpose of the search. Questions to ask include: For what and by whom will the information be used? A topical expert or a novice? Am I looking for a simple fact? A comprehensive overview on the topic? Exploration of a new topic? A systematic review? For the purpose of a systematic review of journal research in the area of health care, PubMed or Medline is the most appropriate retrieval tool to start with, however other databases may be useful ( Table 2 ). In particular, Google Scholar allows one to search the same set of articles as PubMed/MEDLINE, in addition to some from other disciplines, but it lacks a number of key advanced search features that a skilled searcher can exploit in PubMed/MEDLINE.

Examples of Electronic Bibliographic Databases Specific to Health Care

Note: These databases may be available through university or hospital library systems.

An effective way to search the literature is to break the topic into different “building blocks.” The building blocks approach is the most systematic and works the best in periodical databases such as PubMed/MEDLINE. The “blocks” in a “building blocks” strategy consist of the key concepts in the search topic. For example, let’s say we are interested in researching about mobile phone-based interventions for monitoring of patient status or disease management. We could break the topic into the following concepts or blocks: 1. Mobile phones, 2. patient monitoring, and 3. Disease management. Gather synonyms and related terms to represent each concept and match to available subject headings in databases that offer them. Organize the resulting concepts into individual queries. Run the queries and examine your results to find relevant items and suggest query modifications to improve your results. Revise and re-run your strategy based on your observations. Repeat this process until you are satisfied or further modifications produce no improvements. For example in Medline, these terms would be used in this search and combined as follows: cellular phone AND (ambulatory monitoring OR disease management), where each of the key word phrases is an official subject heading in the MEDLINE vocabulary. Keep detailed notes on the literature search, as it will need to be reported in the methods section of the systematic review paper. Careful noting of search strategies also allows you to revisit a topic in the future and confidently replicate the same results, with the addition of those subsequently published on your topic.

3. Assess Quality

There is no consensus on the best way to assess study quality. Many quality assessment tools include issues such as: appropriateness of study design to the research objective, risk of bias, generalizability, statistical issues, quality of the intervention, and quality of reporting. Reporting guidelines for most literature types are available at the EQUATOR Network website ( http://www.equator-network.org/ ). These guidelines are a useful starting point; however they should not be used for assessing study quality.

4. Extract Data and Information

Extract information from each eligible article into a standardized format to permit the findings to be summarized. This will involve building one or more tables. When making tables each row should represent an article and each column a variable. Not all of the information that is extracted into the tables will end up in the paper. All of the information that is extracted from the eligible articles will help you obtain an overview of the topic, however you will want to reserve the use of tables in the literature review paper for the more complex information. All tables should be introduced and discussed in the narrative of the literature review. An example of an evidence summary table is presented in Table 3 .

Example of an evidence summary table

Notes: BP = blood pressure, HbA1c = Hemoglobin A1c, Hypo = hypoglycemic, I = Internet, NS = not significant, PDA = personal digital assistant, QOL = quality of life, SMBG = self-monitored blood glucose, SMS = short message service, V = voice

5. Analyze and Synthesize Data and information

The findings from individual studies are analyzed and synthesized so that the overall effectiveness of the intervention can be determined. It should also be observed at this time if the effect of an intervention is comparable in different studies, participants, and settings.

6. Write the Systematic Review

The PRISMA 12 and ENTREQ 13 checklists can be useful resources when writing a systematic review. These uniform reporting tools focus on how to write coherent and comprehensive reviews that facilitate readers and reviewers in evaluating the relative strengths and weaknesses. A systematic literature review has the same structure as an original research article:

TITLE : The systematic review title should indicate the content. The title should reflect the research question, however it should be a statement and not a question. The research question and the title should have similar key words.

STRUCTURED ABSTRACT: The structured abstract recaps the background, methods, results and conclusion in usually 250 words or less.

INTRODUCTION: The introduction summarizes the topic or problem and specifies the practical significance for the systematic review. The first paragraph or two of the paper should capture the attention of the reader. It might be dramatic, statistical, or descriptive, but above all, it should be interesting and very relevant to the research question. The topic or problem is linked with earlier research through previous attempts to solve the problem. Gaps in the literature regarding research and practice should also be noted. The final sentence of the introduction should clearly state the purpose of the systematic review.

METHODS: The methods provide a specification of the study protocol with enough information so that others can reproduce the results. It is important to include information on the:

• Eligibility criteria for studies: Who are the patients or subjects? What are the study characteristics, interventions, and outcomes? Were there language restrictions?
• Literature search: What databases were searched? Which key search terms were used? Which years were searched?
• Study selection: What was the study selection method? Was the title screened first, followed by the abstract, and finally the full text of the article?
• Data extraction: What data and information will be extracted from the articles?
• Data analysis: What are the statistical methods for handling any quantitative data?

RESULTS: The results should also be well-organized. One way to approach the results is to include information on the:

• Search results: What are the numbers of articles identified, excluded, and ultimately eligible?
• Study characteristics: What are the type and number of subjects? What are the methodological features of the studies?
• Study quality score: What is the overall quality of included studies? Does the quality of the included studies affect the outcome of the results?
• Results of the study: What are the overall results and outcomes? Could the literature be divided into themes or categories?

DISCUSSION: The discussion begins with a nonnumeric summary of the results. Next, gaps in the literature as well as limitations of the included articles are discussed with respect to the impact that they have on the reliability of the results. The final paragraph provides conclusions as well as implications for future research and current practice. For example, questions for future research on this topic are revealed, as well as whether or not practice should change as a result of the review.

REFERENCES: A complete bibliographical list of all journal articles, reports, books, and other media referred to in the systematic review should be included at the end of the paper. Referencing software can facilitate the compilation of citations and is useful in terms of ensuring the reference list is accurate and complete.

The following resources may be helpful when writing a systematic review:

CEBM: Centre for Evidence-based Medicine. Dedicated to the practice, teaching and dissemination of high quality evidence based medicine to improve health care Available at: http://www.cebm.net/ .

CITING MEDICINE: The National Library of Medicine Style Guide for Authors, Editors, and Publishers. This resource provides guidance in compiling, revising, formatting, and setting reference standards. Available at http://www.ncbi.nlm.nih.gov/books/NBK7265/ .

EQUATOR NETWORK: Enhancing the QUAlity and Transparency Of health Research. The EQUATOR Network promotes the transparent and accurate reporting of research studies. Available at: http://www.equator-network.org/ .

ICMJE RECOMMENDATIONS: International Committee of Medical Journal Editors Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals. The ICJME recommendations are followed by a large number of journals. Available at: http://www.icmje.org/about-icmje/faqs/icmje-recommendations/ .

PRISMA STATEMENT: Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Authors can utilize the PRISMA Statement checklist to improve the reporting of systematic reviews and meta-analyses. Available at: http://prisma-statement.org .

THE COCHRANE COLLABORATION: A reliable source for making evidence generated through research useful for informing decisions about health. Available at: http://www.cochrane.org/ .

Examples of Systematic Reviews To Link Research and Quality Improvement

Over the past 17 years more than 300 learners, including physicians, nurses, and health administrators have completed a course as part of a Master of Health Administration or a Master of Science in Health Informatics degree at the University of Missouri. An objective of the course is to educate health informatics and health administration professionals about how to utilize a systematic, scientific, and evidence-based approach to literature searching, appraisal, and synthesis. Learners in the course conduct a systematic review of the literature on a health care topic of their choosing that could suggest quality improvement in their organization. Students select topics that make sense in terms of their core educational competencies and are related to their work. The categories of topics include public health, leadership, information management, health information technology, electronic medical records, telehealth, patient/clinician safety, treatment/screening evaluation cost/finance, human resources, planning and marketing, supply chain, education/training, policies and regulations, access, and satisfaction. Some learners have published their systematic literature reviews 14 – 15 . Qualitative comments from the students indicate that the course is well received and the skills learned in the course are applicable to a variety of health care settings.

Undertaking a literature review includes identification of a topic of interest, searching and retrieving the appropriate literature, assessing quality, extracting data and information, analyzing and synthesizing the findings, and writing a report. A structured step-by-step approach facilitates the development of a complete and informed literature review.

Suzanne Austin Boren, PhD, MHA, (above) is Associate Professor and Director of Academic Programs, and David Moxley, MLIS, is Clinical Instructor and Associate Director of Executive Programs. Both are in the Department of Health Management and Informatics at the University of Missouri School of Medicine.

Contact: ude.iruossim.htlaeh@snerob

None reported.

• Advanced Search

Using Recommendation Systems in Disaster Management: A Systematic Literature Review

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options, recommendations, design and evaluation of a role improvisation exercise for crisis and disaster response teams.

This paper reports a case study, which aimed to investigate what role improvisation plays in crisis and disaster response teams and how improvisation performance of such teams can be simulated in representative contexts and constraints. The case study ...

Dynamic and Time Critical Emergency Management for Level Three Disaster: A Case Study Analysis of Kerala Floods 2018

Extreme variability of weather conditions has contributed to the increase in multi hazards events in Kerala, India. This work analyses the processes, systems and solutions used for management of a 24hr call center for an effective emergency response ...

A Fuzzy Integer Programming Model to Locate Temporary Medical Facilities as Part of Pre-Disaster Management

The number and the scale of natural disasters have drastically increased over the last decades. One of the most vital stages of disaster preparedness is disaster response planning, and it plays an important role in limiting material and immaterial ...

Information

Published in.

Elsevier Science Publishers B. V.

Netherlands

Publication History

Author tags.

• disaster management
• emergency management
• recommendation systems
• systematic literature review
• Research-article

Contributors

Other metrics, bibliometrics, article metrics.

• 0 Total Citations
• 0 Total Downloads
• Downloads (Last 12 months) 0
• Downloads (Last 6 weeks) 0

View options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Open access
• Published: 05 July 2024

Mentoring as a complex adaptive system – a systematic scoping review of prevailing mentoring theories in medical education

• Mac Yu Kai Teo 1 , 2 , 3 ,
• Halah Ibrahim 4 ,
• Casper Keegan Ronggui Lin 1 , 2 , 3 , 5 , 6 ,
• Nur Amira Binte Abdul Hamid 3 ,
• Ranitha Govindasamy 1 , 3 , 6 ,
• Nagavalli Somasundaram 3 , 7 , 8 ,
• Crystal Lim 8 , 9 ,
• Jia Ling Goh 1 , 2 , 3 ,
• Yi Zhou 1 , 2 , 3 ,
• Kuang Teck Tay 1 , 2 , 3 ,
• Ryan Rui Song Ong 1 , 2 , 3 ,
• Vanessa Tan 1 , 2 , 3 ,
• Youru Toh 1 , 2 , 3 ,
• Anushka Pisupati 1 , 2 , 3 ,
• Vijayprasanth Raveendran 1 , 2 , 3 ,
• Keith Zi Yuan Chua 1 , 2 , 3 ,
• Elaine Li Ying Quah 1 , 2 , 3 ,
• Jeevasuba Sivakumar 1 , 2 , 3 ,
• Samyuktha Dhanalakshmi Senthilkumar 1 , 2 , 3 ,
• Keerthana Suresh 1 , 2 , 3 ,
• Wesley Teck Wee Loo 1 , 2 , 3 ,
• Ruth Si Man Wong 1 , 2 , 3 ,
• Yiying Pei 1 , 2 , 3 ,
• Julia Huina Sng 1 , 2 , 3 ,
• Simone Qian Min Quek 1 , 2 , 3 ,
• Jasmine Lerk Juan Owyong 3 ,
• Ting Ting Yeoh 3 , 5 ,
• Eng Koon Ong 2 , 3 , 7 , 10 ,
• Gillian Li Gek Phua 2 , 11 ,
• Stephen Mason 12 ,
• Ruaraidh Hill 13 ,
• Anupama Roy Chowdhury 7 , 14 ,
• Simon Yew Kuang Ong 3 , 7 , 8 &
• Lalit Kumar Radha Krishna 1 , 2 , 3 , 6 , 7 , 12 , 13 , 15  

BMC Medical Education volume  24 , Article number:  726 ( 2024 ) Cite this article

Metrics details

Effective mentorship is an important component of medical education with benefits to all stakeholders. In recent years, conceptualization of mentorship has gone beyond the traditional dyadic experienced mentor-novice mentee relationship to include group and peer mentoring. Existing theories of mentorship do not recognize mentoring’s personalized, evolving, goal-driven, and context-specific nature. Evidencing the limitations of traditional cause-and-effect concepts, the purpose of this review was to systematically search the literature to determine if mentoring can be viewed as a complex adaptive system (CAS).

A systematic scoping review using Krishna’s Systematic Evidence-Based Approach was employed to study medical student and resident accounts of mentoring and CAS in general internal medicine and related subspecialties in articles published between 1 January 2000 and 31 December 2023 in PubMed, Embase, PsycINFO, ERIC, Google Scholar, and Scopus databases. The included articles underwent thematic and content analysis, with the themes identified and combined to create domains, which framed the discussion.

Of 5,704 abstracts reviewed, 134 full-text articles were evaluated, and 216 articles were included. The domains described how mentoring relationships and mentoring approaches embody characteristics of CAS and that mentorship often behaves as a community of practice (CoP). Mentoring’s CAS-like features are displayed through CoPs, with distinct boundaries, a spiral mentoring trajectory, and longitudinal mentoring support and assessment processes.

Recognizing mentorship as a CAS demands the rethinking of the design, support, assessment, and oversight of mentorship and the role of mentors. Further study is required to better assess the mentoring process and to provide optimal training and support to mentors.

Peer Review reports

Effective mentorship during medical training fosters professional development, personal growth, and ethical guidance [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ]. For host institutions, established mentorship programs facilitate knowledge transfer, improve recruitment and retention, and contribute to a culture of continuous learning and growth, ultimately advancing the quality of healthcare delivery and research within the organization [ 1 , 2 , 3 , 4 , 5 , 8 , 9 ]. Yet, despite its importance, medical education still lacks a widely accepted operational definition of mentoring [ 10 ]. Mentorship is often conflated with advising or coaching. While advisors assist trainees in making informed academic decisions and coaches provide training and guidance to help trainees reach specific goals, mentorship is a bidirectional relationship whereby an experienced mentor provides personalized guidance and support to facilitate a mentee’s development [ 11 ]. In recent years, conceptualizations of mentorship have also evolved from this traditional dyadic experienced mentor and novice mentee relationship to peer and group mentoring formats and mentoring networks [ 11 ]. Recent reviews highlight the challenges related to mentoring and attribute multiple ethical issues, including bullying, coercion, misappropriation of mentee funding and resources, and publication parasitism to inadequate structuring, support, and oversight of mentorship programs [ 1 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ]. As accounts of ethical, legal, and professional issues related to mentoring continue to grow, the need for a common understanding and consistent approach to mentoring is evident [ 1 , 18 ].

Current theories of mentoring struggle to contend with mentoring’s personalized, evolving, and reciprocal nature, which is often goal-sensitive and context-specific [ 20 , 21 ]. Several authors have criticized conventional models that do not recognize the dynamic relationship between mentors and mentees and the influence of external factors [ 11 , 12 ]. Some studies suggest that mentoring should be considered a complex adaptive system (CAS) [ 22 , 23 , 24 ]. With such a shift in thinking likely to change the design, support, and oversight of mentoring programs, we evaluate if mentoring displays the characteristics and functions of a CAS to address our primary research question “does mentoring function as a complex adaptive system?”

• Complex adaptive systems

Some authors propose that a CAS-led perspective better captures mentoring’s non-linear, diverse, individualized, and unpredictable interrelationships [ 25 ]. A CAS is a system composed of many interacting and interdependent components (agents), whereby one agent’s actions can change the context for the others [ 26 ]. Features of CAS include complexity, adaptation, non-linearity, and self-organization, resulting in the spontaneous emergence of new and unpredictable patterns, behaviors, and trajectories. We define these features throughout the manuscript and summarize the terms in Table  1 as characterized by Ellis et al. [ 27 ] and Gear et al. [ 28 ].

Theoretical lens

A mentoring ecosystem encompasses a broad range of mentors, mentees, and stakeholders, including institutions, all contributing to individual growth and development through mentorship. The concept of the mentoring ecosystem is like a Community of Practice, or a social network with mutual experiences and values [ 29 ], and is shaped around a predetermined course from marginal participation at the periphery of the mentoring program to a more central role within the mentoring program [ 29 ] (Fig.  1 ). This mentoring trajectory is framed on mentoring stages [ 30 , 31 ], or clearly delineated phases of the mentoring process. Transitions from one stage to another create ideal assessment points, which in turn, inform the longitudinal mentoring support system, or mentoring umbrella. The mentoring umbrella is a framework where multiple forms of mentoring and support, including supervision, coaching, tutoring, instruction, and teaching, are provided to support an individual’s growth and development, like how an umbrella provides protection and coverage [ 32 ]. This approach ensures that mentees receive comprehensive support from different sources to enhance their learning, skill development, and career advancement [ 32 ]. The combination of the mentoring trajectory and mentoring umbrella creates the mentoring tube, which guides mentoring progress.

Mentoring ecosystem. The yellow circles represent the mentee’s microenvironment while the blue circles symbolize other stakeholders’ microenvironments. The dark green spiral represents the mentoring tube, and the thin blue lines represent the changing course of the mentoring relationship along the mentoring trajectory. The mentoring trajectory is framed around key stages of development. Some of these stages are highlighted. The mentoring trajectory is not depicted as a smooth course underlining the inevitable changes apparent across the mentoring stages

SEBA review methodology

Using Krishna’s Systematic Evidence-Based Approach (SEBA) to guide our scoping review [ 32 , 33 , 34 , 35 ], we explore mentoring in medical education as a sociocultural construct shaped by multiple stakeholder and host organization-related factors. This approach also accommodates the CAS lens through which we evaluate the different aspects of mentoring for features of CAS. SEBA is a methodologic framework for conducting systematic scoping reviews. The steps of the SEBA process involve: (1) a systematic approach whereby teams of medical education experts and researchers agree upon the research questions, search terms, and databases to be included; (2) the split approach in which a research team conducts inductive thematic analysis of the included articles allowing themes to emerge from the data while other research team(s) independently use a predefined set of codes to guide the analysis and identify themes; (3) the jigsaw perspective involves combining overlapping and complementary themes to create larger categories of themes; (4) a comparison process with the features of CAS ensures that relevant themes are not omitted; (5) analysis of data and non-data driven literature compares the themes derived from evidenced-based publications with those from non-data-based articles (editorials, grey literature, letters, opinion pieces, and perspectives) for similarity to ensure that the non-data-based articles do not bias the analysis; and (6) synthesis where the derived themes create the domains that inform the discussion (Fig.  2 ).

The SEBA process

Reflexivity

The research team consisted of medical students and research assistants, guided by Internal Medicine and Palliative Care consultants, with expertise in medical education, qualitative analysis, and conducting systematic reviews. The medical students were members of a peer-mentorship research program; their personal experiences influenced the study design and data interpretation. To provide a balanced review, an expert team comprising of a librarian from the National University of Singapore’s (NUS) Yong Loo Lin School of Medicine (YLLSoM) and local educational experts and clinicians at YLLSoM, National Cancer Centre Singapore, Palliative Care Institute Liverpool, and Duke-NUS Medical School guided the 6-stages of the SEBA process. The teams also engaged in personal and group reflexivity throughout the process to minimize the impact of personal experience bias.

Stage 1: systematic approach

This SEBA-systematic scoping review is guided by the PRISMA-P checklist to ensure a reproducible and robust mapping of current notions of mentoring.

Identifying the research question

Guided by the expert team, the research team determined the primary research question to be: “does mentoring function as a complex adaptive system?” The secondary research question is: “what characteristics of CAS are evident in mentoring?”

Inclusion criteria

A population, concept, context (PCC) study design format was adopted to guide the research [ 36 ] (Table  2 ). We included all study types (quantitative, qualitative, mixed methods) and non-empirical manuscripts (perspectives, editorials, letters) involving medical students and medical trainees and physicians in Internal Medicine and its related subspecialties. We excluded studies from other disciplines and those involving mentorship by patients or interdisciplinary mentors, along with studies dealing with supervision, coaching, role-modeling, advising, or sponsorship. In keeping with Pham et al.’s [ 37 ] recommendations on sustaining the research process and accommodating existing manpower and time constraints, the research team restricted the searches to articles published between 1st January 2000 and 31st December 2023.

Database searching

Eleven members of the research teams searched five bibliographic databases (PubMed, Embase, PsycINFO, Cochrane Database of Systematic Reviews, CINAHL, Scopus) between 13 February 2023 and 20 April 2024 (Table  3 ). The research teams, each comprised of medical students and a senior reviewer, independently carried out the searches. The search terms and strategies used for database searching are detailed in Appendix  1 .

Extracting and charting the data

Each group independently reviewed the abstracts and titles and discussed their findings at online meetings where Sandelowski and Barroso [ 148 ]’s ‘negotiated consensual validation’ was used to achieve consensus on the final list of full-text articles to be reviewed. Data extracted from each manuscript meeting inclusion criteria included the author, year of publication, study type, study population, study location, components of the mentorship ecosystem, including mentoring approach/theories, stakeholders, mentoring structure and relationships, environment and external influences, and main findings of the study. The characteristics of all included studies are listed in Appendix  1 .

Stage 2: Split approach

Krishna’s ‘Split Approach’ [ 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 , 98 , 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 , 139 , 140 , 141 , 142 , 143 , 144 , 145 , 146 , 147 , 148 , 149 , 150 , 151 ] was employed to enhance the reliability of the data analyses. This approach involves the research team dividing (or ‘splitting’) into different groups to independently analyze the manuscripts. This concurrent analysis enables review of data from different perspectives while also aiming to reduce omission of new findings or negative reports. For this review, three groups of researchers independently analyzed the included articles. Using best practices, the first team summarized and tabulated the included full-text articles [ 152 , 153 ] (Appendix  1 ). Concurrently, the second team analyzed the included articles using Braun and Clarke’s approach to thematic analysis [ 154 ]. A third team of researchers employed Hsieh and Shannon’s [ 155 ] approach to directed content analysis to analyze the included articles using pre-determined codes drawn from several published manuscripts on mentorship in medical education [ 7 , 156 , 157 ]. These studies were chosen because they provide the most recent review of mentoring practice at the time of this review [ 7 ] and offer the most recent longitudinal work on the subject [ 156 , 157 ].

Stage 3: Jigsaw perspective

In keeping with SEBA’s iterative process, the themes were reviewed by the expert and research teams. Overlaps between the themes were viewed as pieces of a jigsaw puzzle with the intention of combining overlapping or complementary pieces to create a bigger piece of the puzzle to form larger categories of themes.

Stage 4: Comparison

Comparisons of the themes were made with the features of CAS identified by Ellis et al. [ 27 ] and Gear et al. [ 28 ], specifically multiple agents, self-organizing networks, co-evolution, system adaptation, non-linearity, feedback loops, emergence, boundaries, path dependency, and ‘far from equilibrium (Table  1 ). This step ensures that important themes were not omitted.

The PRISMA diagram illustrates the process (Fig.  3 ). Of 5,704 abstracts reviewed, 134 full-text articles were evaluated, and 216 articles were included (additional articles included following snowballing, or review of the references of included articles). The themes elicited during thematic analysis of all 216 manuscripts were overlapped and combined (jigsaw approach) into larger categories of themes and compared with features of CAS to create 2 domains, namely mentoring relationships and mentoring programs, each with sub-themes as detailed below.

PRISMA flowchart. Snowballing articles were derived from searching the references of all included articles

Domain 1. Mentoring relationships

Mentoring relationships are influenced by the various stakeholders (agents) and the mentoring process.

Multiple stakeholders (agents)

A key feature of CAS is the presence of multiple agents interacting within collaborative networks [ 27 , 28 , 30 , 158 , 159 , 160 ]. Our results support that mentors, mentees, institutional leaders, and multiple other stakeholders interact within the mentorship ecosystem by exchanging resources and information, thereby influencing each other’s perspectives and behaviors and collectively shaping the trajectory and outcomes of the mentorship dynamic. Several authors explored the roles of mentors, peer-mentors, mentees, and the host organization in mentoring programs and noted that the nature of collaborative networks can be a mix of formal and informal approaches [ 7 , 161 , 162 , 163 , 164 , 165 , 166 ]. Even within formal programs [ 30 , 167 , 168 , 169 , 170 ], with their clearly defined roles and responsibilities, expectations, and codes of conduct, the presence of multiple agents, each with their roles, goals, responsibilities, and areas of interest, suggests that interactions may veer toward the informal. Such variation draws attention shifting nature of the relationships between different stakeholders throughout a mentorship.

Similarly, stakeholder influence on mentees varies according to the circumstances and time. Some studies described fluidity in the nature of interactions between stakeholders, suggesting that membership in the mentoring relationship can change [ 7 , 34 , 157 , 163 ]. New members may replace those who have completed their respective tasks, while other members may leave and re-enter the mentoring program at different points with different roles and responsibilities and varying levels of participation. Re-forming and adjusting mentoring relationships between new and returning stakeholders introduces more complexity. In addition, medical education’s hierarchical nature also impacts mentoring interactions and relationships, particularly when considering evolving circumstances and changing goals, expectations, and timelines.

The presence of multiple agents highlights the bilateral, but not necessarily equal, impact that mentoring relationships have on individual members. Stakeholder views and responses to their mentoring experiences are influenced by multiple factors, including their personal belief systems, developing competencies, coping strategies, psycho-emotional state, and maturing relationships with other stakeholders. Concurrently, the agent’s conduct, actions, and motivations are also influenced by contextual considerations, including changes to their professional roles and responsibilities, as well as stage-specific modifications to the curricula, host organization-related factors, mentoring culture, and access to support (Table  3 ).

Mentoring structures

The process, or structures (7, 49, 56, 60–63), of mentoring play a key role in shaping mentoring relationships [ 168 , 171 , 172 ]. We use Krishna et al.’s [ 6 ] concept of the mentoring ecosystem to illustrate the role that mentoring stages, mentoring trajectory, mentoring environment, mentoring umbrella, and the mentoring approach have on the CAS-related concepts of path dependency, boundaries, and adaptations in mentoring relationships [ 47 ].

Path dependency

Current concepts of path dependency [ 165 , 173 , 174 ] focus on the impact of past experiences or trajectories [ 7 ] on the current and future state of the system, suggesting that the cumulative effects of individual and programmatic experiences within the system have an enduring impact on its current structure and future potential [ 27 , 28 , 159 ]. Path dependency acknowledges that previous mentorship experiences [ 158 , 175 ], historical choices [ 170 ] made by mentors and mentees, and the development of mentoring structures can shape the current dynamics and long-term possibilities of mentoring relationships. The impact of many of these effects is managed through the alignment of expectations [ 171 ] and available support.

Boundaries in CAS represent sociocultural constructs that highlight points of contact with other entities. Mentorship programs often span multiple levels of organization, including individual, interpersonal, institutional, and societal levels. These ‘fuzzy boundaries’ surrounding the micro-environments of each stakeholder [ 176 ] underscore the connections that influence the environment and adjacent micro-environments, adding another layer of complexity to the system and influencing the types of outcomes that can be achieved. The impact of these changes on the micro-environment depends on various factors, including the nature and duration of the mentoring relationship, the seniority, roles [ 166 ], and motivations of the stakeholders, the roles and expectations [ 170 , 177 ] within the specific stages of mentoring, and the support provided by the mentoring umbrella [ 33 , 35 , 157 , 178 ]. Moreover, the ‘fuzzy boundaries’ also enable the mentoring umbrella to shape micro-environments by providing timely and appropriate feedback, support, and remediation along the mentoring trajectory.

System adaptation

System adaptation refers to a system’s ability to modify itself to maintain stability, optimize performance, or achieve objectives despite disturbances [ 27 , 28 ]. Within CAS, adaptations [ 179 ] are made to avoid major changes [ 180 ] to the system. Here, adaptability hinges on finding a balance between flexibility and consistency [ 7 ], focusing on making the smallest possible changes to the least significant elements to facilitate meaningful changes within the evolving micro-environments along the mentoring trajectory. In the mentoring ecosystem [ 181 ], the aim is to prioritize changes at the individual level and among a select few stakeholders [ 158 , 176 ] to preserve stability in the broader program [ 182 ] while nurturing the mentorship process.

Co-evolution

Within CAS, interactions between agents give rise to mutual transformations [ 171 , 183 , 184 ]. As mentees adapt to the different goals, roles, and situated learning requirements across distinct mentoring stages along the mentoring trajectory, their interactions with other stakeholders [ 163 ] lead to changes in these stakeholders. This co-evolution is focused on preserving the quality of interactions within their mentoring relationship, referred to as mentoring dynamics. These dynamics are pivotal in shaping the personalized and enduring mentoring relationships that underpin mentoring success [ 6 , 185 , 186 ]. For example, as a mentee gains confidence and skills through effective feedback, the mentor can gain valuable insights to improve communication and feedback practices.

Feedback loops

Reflections on new mentoring experiences [ 171 , 180 ], adaptations, and co-evolutions serve to reinforce positive changes while mitigating the repercussions of negative changes. This recursive influence of feedback loops also extends to thought processes, decision-making, and future actions [ 33 , 35 , 157 , 168 , 178 , 187 ].

The processes of adaptation [ 176 ], co-evolution, and feedback loops [ 163 ] give rise to the concept of ‘emergent behavior,’ behavior that emerges from interactions within the system, often focused on sustaining specific goals [ 27 , 28 ]. ‘Emergent behaviour’ is shaped by the prevailing conditions, available resources and options, guidance received, and stakeholder interpretation of unfolding events.

Self-organization

When individuals experience shifts in their attitudes, thinking [ 32 , 188 ], practice, and belief systems in response to ongoing changes, feedback, and environmental shifts, self-organization occurs. The mentoring framework, development of competencies [ 189 ], coping strategies, meaning-making process, and psycho-emotional state of individuals are pivotal in shaping self-organization within their micro-environment. These transformations in thoughts, emotions, and practices are guided by regnant standards [ 1 , 7 , 151 , 190 , 191 , 192 , 193 , 194 , 195 , 196 , 197 , 198 , 199 , 200 , 201 ].

Self-organization subsequently influences the mentoring trajectory. When these changes align with mentoring objectives and approach, and are consistent with the overall trajectory, they facilitate goal attainment. However, if the trajectory deviates from alignment with these elements, mentees may struggle to reach their goals.

Non-linearity

This evolving membership [ 158 , 171 , 180 , 202 ] of mentoring programs set within a hierarchical environment characterized by differences in diversity [ 174 , 203 , 204 , 205 , 206 ], gender [ 206 , 207 ], seniority, roles, and responsibilities [ 7 , 163 ], coupled with adaptations, co-evolution, and the emergence of new behaviours, lead to non-linear responses [ 32 , 164 , 170 ] in interactions among stakeholders with diverse roles and responsibilities. This non-linearity [ 156 , 159 , 163 , 208 ] is also apparent in the way individuals respond to various stimuli [ 1 , 18 , 33 , 35 , 95 , 96 , 97 , 98 , 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 , 139 , 140 , 141 , 142 , 143 , 144 , 145 , 146 , 147 , 148 , 149 , 150 , 151 , 152 , 153 , 154 , 155 , 156 , 157 , 158 , 159 , 160 , 161 , 162 , 163 , 164 , 165 , 166 , 167 , 168 , 169 , 170 , 171 , 172 , 173 , 174 , 175 , 176 , 177 , 178 , 179 , 180 , 181 , 182 , 183 , 184 , 185 , 186 , 187 , 188 , 189 , 190 , 191 , 192 , 193 , 194 , 195 , 196 , 197 , 198 , 199 , 200 , 201 , 202 , 203 , 204 , 205 , 206 , 207 , 208 , 209 , 210 ].

Far from equilibrium

The evolving processes [ 173 ] of adaptations [ 211 ], co-evolution, emergent behaviour, self-organization, non-linearity, and the influence of feedback loops [ 163 , 187 ] expose a system in a state ‘far from equilibrium’ [ 27 , 28 ], highlighted further during the COVID-19 pandemic [ 158 , 212 , 213 ] .In this context, even minor changes can lead to disproportionate impacts on mentoring relationships and processes [ 1 , 7 , 18 , 21 , 29 , 35 , 95 , 156 , 198 , 210 , 214 , 215 , 216 , 217 , 218 , 219 , 220 , 221 ]. In mentorship, being ‘far from equilibrium’ can also suggest a state of continuous learning, growth, and innovation, where mentor and mentees interactions are dynamic, challenging existing norms and practices and creating new possibilities for personal and professional development.

Domain 2. Mentoring programs

Mentoring programs [ 30 , 169 , 179 , 210 , 212 , 222 , 223 , 224 , 225 , 226 , 227 ] are often integrated [ 188 , 208 , 228 ] within the formal curriculum [ 7 ] and subject to oversight [ 176 , 229 ] and evaluation by the host organization [ 53 , 156 , 230 ]. The increased emphasis on oversight within mentorship has grown amidst mounting concerns about ethical issues in mentoring [ 1 , 12 , 14 , 15 , 16 , 17 , 18 , 19 ]. While mentoring programs allow for a degree of flexibility, adaptability, and responsiveness, these functions are constrained by host organization-related factors and standards [ 231 ]. There are concerted efforts to instil consistency into practice, as evident in the structuring of the mentoring trajectory through the mentoring framework, the personalization of mentoring experiences, support, and assessments [ 189 ], and the establishment of clear transition points from one mentoring stage to the next, ensuring that the required knowledge, skills [ 232 ], and competencies for progression have been acquired. Furthermore, there is an emphasis on establishing clear expectations [ 233 , 234 ] for the roles and responsibilities of stakeholders at each stage, particularly in light of their differing roles along the mentoring trajectory and maintaining clear standards for their engagement [ 202 ] as some stakeholders move in and out of various mentoring stages.

Moreover, mentoring programs can also establish clear areas of interest, goals, expectations, timelines, and entry criteria [ 210 ]. The mentoring project, setting, structure, and the faculty involved also help distinguish the mentoring process from other mentoring projects [ 38 ]. For example, the Palliative Medicine Initiative (PMI), as described by Krishna and colleagues, represents a structured research mentoring program jointly established within the Division of Supportive and Palliative Care and the Centre for Biomedical Ethics at the Yong Loo Lin School of Medicine at the University of Singapore. Comprised mainly of palliative care physicians and ethicists as mentors, the PMI is framed as a unique research-oriented mentoring program, with a specific focus on ethics and palliative care research.

Several authors frame mentoring projects in medical education as a community of practice (CoP). In CoPs, learning is a collaborative and social process. The unique nature of each mentoring project, with its specific inclusion and exclusion criteria [ 188 ], focal points, approaches, and distinct project boundaries underlie the notion that each project functions as a CoP. As most programs host more than one mentoring project, the mentoring program can be viewed as a landscape of practice (LoP), a complex tapestry of various CoPs [ 199 , 235 , 236 , 237 , 238 , 239 , 240 ]. This view of mentoring programs as LoPs is supported by recent studies [ 156 , 157 ], which reveal complexities within the mentoring process that arise when members of the host organization, mentors, or peer mentors engage in more than one project or CoP simultaneously, leading to a situation in which events or adaptations in one project can not only affect stakeholders in other CoPs but, perhaps more importantly, can also impact the nature and quality of their mentoring relationships in other projects within the LoP [ 156 ].

Stage 5: analysis of evidence-based and non-data driven literature

Evidenced-based data from bibliographic databases were separated from grey literature and opinion, perspectives, editorials, letters, and non-data-based articles drawn from bibliographic databases and both groups were thematically analysed separately. The themes from both groups were compared to determine if there were additional themes in the non-data-driven group that could influence the narrative. There was consensus from the research team that themes from non-data-driven and peer-reviewed evidence-based publications were similar and did not bias the analysis.

Stage 6: synthesis

In answering the research question “what characteristics of CAS are evident in mentoring?”, this review highlights how mentoring relationships involve multiple stakeholders and exhibit CAS-like features, such as emergence, adaptability, self-organization, co-evolution, non-linearity, path dependency, and a state far-from-equilibrium. These dynamics highlight the unpredictable and nonlinear nature of mentorship. However, traditional views of mentoring relationships impose rigid boundaries and predefined trajectories, akin to CoPs, which can stifle the natural evolution and complexity of mentoring relationships. The data challenge us to rethink how we define and approach mentorship, emphasizing the need to embrace its adaptability and self-organizing nature. They suggest that by acknowledging and leveraging the CAS-like characteristics of mentorship, we can foster more innovative and effective mentoring processes.

Our findings also emphasize that efforts to guide the mentoring process can occur at all stages of this journey. This is evident in how emergent behavior, adaptations, co-evolution, and self-organization are influenced by many host organization-related factors and are incorporated within professional codes of conduct, ethical standards, and organizational expectations. These features underscore the need for a more nuanced CAS-based theory to describe mentoring relationships and the factors that impact them. Adapting a resilience framework, or a model that emphasizes the capacity of systems to absorb disturbances, adapt to changing conditions, and maintain core functions [ 241 ], for example, to the context of CAS and mentorship in medical education can provide a different perspective on the dynamic and non-linear nature of mentorship relationships and how they can be influenced by factors such as feedback loops, emergence, self-organization, and adaptation. Ultimately, this can promote more supportive and sustainable approaches to mentoring to better address the diverse and evolving needs of mentees.

Our findings have several curricular and policy implications. First, given the multiple agents involved and the unpredictable nature of mentoring relationships [ 27 , 28 ], mentoring programs should be embedded within a formal framework. This structure allows the host organization to establish clear guidelines and align expectations, timelines, and support. Moreover, it promotes consensus on the mentoring approach, structure, trajectory, and stages. Within this formal structure, accessible, longitudinal training opportunities should be established. Communication, assessment, and support systems for all stakeholders can also be put in place to create an environment that is conducive to effective mentoring, thereby minimizing path dependency, or the impact of historical decisions or biases [ 27 , 28 ]. As mentorship relationships are non-linear whereby small changes can have a disproportionate impact on the mentee’s development and career trajectory [ 27 , 28 ], the host organization should take an active role in supporting mentor training and conducting regular assessments of individual projects and the program as a whole. This is particularly important as mentoring is recognized as a proactive process that relies on the involvement of mentoring faculty. This also acknowledges co-evolution in mentorship and supports bidirectional growth and learning between mentors and mentees. Formal recognition of mentor contributions is warranted, along with the allocation of protected time from clinical service to ensure that mentors can effectively meet their mentoring obligations.

To mitigate the risks and biases of path dependency [ 27 , 28 ], host organizations should evaluate institutional practices that may be influenced by historical factors and biases and conduct comprehensive, longitudinal assessments of the stakeholders, their mentoring relationships, progress, development, the program environment, structure, and approach. As part of this process, the use of mentoring diaries can provide a better understanding of mentee and mentor experiences over time and changing situations. Additionally, mentoring portfolios can provide multisource feedback and evidence of research, clinical, and professional development.

Limitations

Analysis of literature on mentoring programs in medical schools is largely drawn from North American and European practices, possibly limiting generalizability to non-Western settings. We limited the search to studies involving medical students and residents in internal medicine and related sub-specialties. Mentoring experiences of surgical and other non-medical specialty residents may be different. While introducing perspective data gives insights into the initiation and development of mentoring programs, selection or reporting bias may be introduced. Further, the applicability of the findings in other healthcare settings may be compromised by conflations of mentoring in clinical and non-clinical settings.

The literature supports the resemblance between mentorship and complex adaptive systems, highlighting the dynamic, emergent, and nonlinear nature of mentoring relationships, while advocating for a paradigm shift towards more supportive and efficient mentorship practices in medical education. Further study of the environments and boundaries of mentoring relationships are needed to guide our evolving perspective of mentoring.

Data availability

The datasets supporting the conclusions of this article are included within the article and its additional files.

Abbreviations

Complex Adaptive System

Systematic Evidence-Based Approach

Population, Concept Context

Professional Identity Formation

Community of Practice

Landscape of Practice

Cheong CWS, Chia EWY, Tay KT, Chua WJ, Lee FQH, Koh EYH, et al. A systematic scoping review of ethical issues in mentoring in internal medicine, family medicine and academic medicine. Adv Health Sci Educ Theory Pract. 2020;25(2):415–39.

Article   Google Scholar  

Choo Hwee P, Hwee Sing K, Yong Hwang MK, Hum Yin Mei A. A qualitative study on the experiences and reflections of junior doctors during a palliative care rotation: perceptions of challenges and lessons learnt. J Pain Symptom Manage. 2020;60(3):549–e558541.

Aslakson R, Kamal A, Gelfman L, Mazanec P, Morrison RS, Ferrell B, et al. Are you my mentor? A panel discussion featuring an all-star cast of AAHPM and HPNA mentors and mentees (th319). J Pain Symptom Manage. 2015;49(2):342–3.

Zhukovsky DS, Bruera E, Rajagopal MR, Rodin G. B07-B building the future of palliative care: Mentoring our people. J Pain Symptom Manage. 2016;52(6):e18–9.

Defilippi KM, Cameron S. Promoting the integration of quality palliative care: the South African mentorship program. J Pain Symptom Manage. 2007;33(5):552–7.

Krishna LKR, Tan LHE, Ong YT, Tay KT, Hee JM, Chiam M, et al. Enhancing mentoring in palliative care: an evidence based mentoring framework. J Med Educ Curric Dev. 2020;7:2382120520957649–2382120520957649.

Goh S, Wong RSM, Quah ELY, Chua KZY, Lim WQ, Ng ADR et al. Mentoring in palliative medicine in the time of COVID-19: A systematic scoping review. BMC Med Educ. 2022;22(1):Article 359.

Carey EC, Weissman DE. Understanding and finding mentorship: a review for junior faculty. J Palliat Med. 2010;13(11):1373–9.

Kashiwagi DT, Varkey P, Cook DA. Mentoring programs for physicians in academic medicine: a systematic review. Acad Med. 2013;88(7):1029–37.

Jacobi M. Mentoring and undergraduate academic success: a literature review. Rev Educ Res. 1991;61(4):505–32.

Ramani S, Kusurkar RA, Lyon-Maris J, Pyorala E, Rogers GD, Samarasekera DD et al. Mentorship in health professions education - an AMEE guide for mentors and mentees: AMEE guide 167. Med Teach. 2023:1–13.

Singh T, Singh A. Abusive culture in medical education: mentors must mend their ways. J Anaesthesiol Clin Pharmacol. 2018;34(2):145–7.

Mahmoudi M. The need for a global committee on academic behaviour ethics. Lancet. 2019;394(10207):1410.

Abu-Zaid A. Protecting medical students against workplace research bullying: a graduate’s experience and standpoint. Educ Health (Abingdon). 2020;33(2):79–80.

Olaussen A, Reddy P, Irvine S, Williams B. Peer-assisted learning: time for nomenclature clarification. Med Educ Online. 2016;21(1):30974–8.

Gray K, Annabell L, Kennedy G. Medical students’ use of Facebook to support learning: insights from four case studies. Med Teach. 2010;32(12):971–6.

Klarare A, Hansson J, Fossum B, Fürst CJ, Lundh Hagelin C. Team type, team maturity and team effectiveness in specialist palliative home care: an exploratory questionnaire study. J Interprof Care. 2019;33(5):504–11.

Kow CS, Teo YH, Teo YN, Chua KZY, Quah ELY, Kamal NHBA, et al. A systematic scoping review of ethical issues in mentoring in medical schools. BMC Med Educ. 2020;20(1):Article246.

Kwok LS. The white bull effect: abusive coauthorship and publication parasitism. J Med Ethics. 2005;31(9):554–6.

Yeam CT, Wesley LTW, Margaret EHF, Kanesvaran R, Krishna LKR. An evidence-based evaluation of prevailing learning theories on mentoring in palliative medicine. Palliat Med Care. 2016;3(1):1–7.

Google Scholar  

Wesley L, Ikbal M, Wu J, Wahab M, Yeam C. Towards a practice guided evidence based theory of mentoring in palliative care. J Palliat Care Med. 2017;7(296):2.

Jones R, Brown D. The mentoring relationship as a complex adaptive system: finding a model for our experience. Mentor Tutoring: Partnersh Learn. 2011;19(4):401–18.

Jones R, Corner J. Seeing the forest and the trees: a complex adaptive systems lens for mentoring. Hum Relat. 2012;65(3):391–411.

Woodruff JN. Accounting for complexity in medical education: a model of adaptive behaviour in medicine. Med Educ. 2019;53(9):861–73.

Smith PA, Palmberg K. Complex adaptive systems as metaphors for organizational management. Learn Organ. 2009;6(16).

Mennin S. Self-organisation, integration and curriculum in the complex world of medical education. Med Educ. 2010;44(1):20–30.

Ellis B, Herbert S. Complex adaptive systems (cas): an overview of key elements, characteristics and application to management theory. Inf Prim Care. 2011;19:33–7.

Gear C, Eppel E, Koziol-McLain J. Advancing complexity theory as a qualitative research methodology. Int J Qual Methods. 2018;17:1–10.

Cruess RL, Cruess SR, Steinert Y. Medicine as a community of practice: implications for medical education. Acad Med. 2018;93(2):185–91.

Haas DM, Hadaie B, Ramirez M, Shanks AL, Scott NP. Resident research mentoring teams: a support program to increase resident research productivity. J Grad Med Educ. 2023;15(3):365–72.

Ravishankar M, Areti A, Kumar VR, Rani TS, Ananthakrishnan P. Competency-based postgraduate training: mentoring and monitoring using entrustable professional activities with descriptive rubrics for objectivity— a step beyond Dreyfus. Natl Med J India. 2024;36(3):176–81.

Toh RQE, Koh KK, Lua JK, Wong RSM, Quah ELY, Panda A et al. The role of mentoring, supervision, coaching, teaching and instruction on professional identity formation: A systematic scoping review. BMC Med Educ. 2022;22(1):Article 531.

Lim JY, Ong SYK, Ng CYH, Chan KLE, Wu SYEA, So WZ et al. A systematic scoping review of reflective writing in medical education. BMC Med Educ. 2023;23(1):Article 12.

Tay KT, Tan XH, Tan LHE, Vythilingam D, Chin AMC, Loh V, et al. A systematic scoping review and thematic analysis of interprofessional mentoring in medicine from 2000 to 2019. J Interprof Care. 2021;35(6):927–39.

Teo KJH, Teo MYK, Pisupati A, Ong RSR, Goh CK, Seah CHX, et al. Assessing professional identity formation (PIF) amongst medical students in oncology and palliative medicine postings: a SEBA guided scoping review. BMC Palliat Care. 2022;21(1):Article200.

Osama T, Brindley D, Majeed A, Murray KA, Shah H, Toumazos M, et al. Teaching the relationship between health and climate change: a systematic scoping review protocol. BMJ Open. 2018;8(5):e020330.

Pham MT, Rajic A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5(4):371–85.

Keating JA, Jasper A, Musuuza J, Templeton K, Safdar N. Supporting midcareer women faculty in academic medicine through mentorship and sponsorship. J Contin Educ Health Prof. 2022;42(3):197–203.

Crampton PES, Afzali Y. Professional identity formation, intersectionality and equity in medical education. Med Educ. 2021;55(2):140–2.

Wyatt TR, Balmer D, Rockich-Winston N, Chow CJ, Richards J, Zaidi Z. Whispers and shadows’: a critical review of the professional identity literature with respect to minority physicians. Med Educ. 2021;55(2):148–58.

Chow CJ, Byington CL, Olson LM, Ramirez KPG, Zeng S, López AM. A conceptual model for understanding academic physicians’ performances of identity: findings from the University of Utah. Acad Med. 2018;93(10):1539–49.

Ong E, Krishna L. Perspective from Singapore. NUS; 2014.

Ong E, Krishna L, Neo P. The sociocultural and ethical issues behind the decision for artificial hydration in a young palliative patient with recurrent intestinal obstruction. Ethics Med. 2015;31(1):39–51.

Surbone A, Baider L. Personal values and cultural diversity. J Med Person. 2013;11(1):11–8.

Au A. Online physicians, offline patients. Int J Sociol Soc Policy. 2018;38(5–6):474–83.

Al-Abdulrazzaq D, Al-Fadhli A, Arshad A. Advanced medical students’ experiences and views on professionalism at Kuwait University. BMC Med Educ. 2014;14(1):Article150.

Kavas MV, Demirören M, Koşan AMA, Karahan ST, Yalim NY. Turkish students’ perceptions of professionalism at the beginning and at the end of medical education: a cross-sectional qualitative study. Med Educ Online. 2015;20(1):26614.

Byszewski A, Hendelman W, McGuinty C, Moineau G. Wanted: role models - medical students’ perceptions of professionalism. BMC Med Educ. 2012;12(1):115.

Smith SE, Tallentire VR, Cameron HS, Wood SM. The effects of contributing to patient care on medical students’ workplace learning. Med Educ. 2013;47(12):1184–96.

Kenny NP, Mann KV, MacLeod H. Role modeling in physicians’ professional formation: reconsidering an essential but untapped educational strategy. Acad Med. 2003;78(12):1203–10.

Rosenblum ND, Kluijtmans M, Ten Cate O. Professional identity formation and the clinician–scientist: a paradigm for a clinical career combining two distinct disciplines. Acad Med. 2016;91(12):1612–7.

Meyer EM, Zapatka S, Brienza RS. The development of professional identity and the formation of teams in the Veterans Affairs Connecticut Healthcare System’s Center of Excellence in Primary Care Education Program (CoEPCE). Acad Med. 2015;90(6):802–9.

Birden H, Glass N, Wilson I, Harrison M, Usherwood T, Nass D. Teaching professionalism in medical education: a best evidence medical education (BEME) systematic review. BEME Guide 25 Med Teach. 2013;35(7):e1252–1266.

Gilligan C, Loda T, Junne F, Zipfel S, Kelly B, Horton G, et al. Medical identity; perspectives of students from two countries. BMC Med Educ. 2020;20(1):Article420.

Wang XM, Swinton M, You JJ. Medical students’ experiences with goals of care discussions and their impact on professional identity formation. Med Educ. 2019;53(12):1230–42.

Witman Y. What do we transfer in case discussions? The hidden curriculum in medicine…. Perspect Med Educ. 2014;3(2):113–123.

Hafferty FW, Franks R. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69(11):861–71.

Whitehead C, Kuper A, Freeman R, Grundland B, Webster F. Compassionate care? A critical discourse analysis of accreditation standards. Med Educ. 2014;48(6):632–43.

Seoane L, Tompkins LM, De Conciliis A, Boysen PG. Virtues education in medical school: the foundation for professional formation. Ochsner J. 2016;16(1):50–5.

Gaufberg E, Bor D, Dinardo P, Krupat E, Pine E, Ogur B, et al. In pursuit of educational integrity: Professional identity formation in the Harvard Medical School Cambridge Integrated Clerkship. Perspect Biol Med. 2017;60(2):258–74.

Monrouxe LV. Identity, identification and medical education: why should we care? Med Educ. 2010;44(1):40–9.

Chuang AW, Nuthalapaty FS, Casey PM, Kaczmarczyk JM, Cullimore AJ, Dalrymple JL et al. To the point: Reviews in medical education - taking control of the hidden curriculum. Am J Obstet Gynecol. 2010;203(4):316.e311-316.e316.

Kay D, Berry A, Coles NA. What experiences in medical school trigger professional identity development? Teach Learn Med. 2019;31(1):17–25.

Jarvis-Selinger S, Pratt DD, Regehr G. Competency is not enough: integrating identity formation into the medical education discourse. Acad Med. 2012;87(9):1185–90.

Frost HD, Regehr G. I am a doctor: negotiating the discourses of standardization and diversity in professional identity construction. Acad Med. 2013;88(10):1570–7.

MacLeod A. Caring, competence and professional identities in medical education. Adv Health Sci Educ Theory Prac. 2011;16(3):375–94.

Rodríguez C, López-Roig S, Pawlikowska T, Schweyer FX, Bélanger E, Pastor-Mira MA, et al. The influence of academic discourses on medical students’ identification with the discipline of family medicine. Acad Med. 2015;90(5):660–70.

Warmington S, McColl G. Medical student stories of participation in patient care-related activities: the construction of relational identity. Adv Health Sci Educ Theory Prac. 2017;22(1):147–63.

Foster K, Roberts C. The heroic and the villainous: a qualitative study characterising the role models that shaped senior doctors’ professional identity. BMC Med Educ. 2016;16(1):Article206.

Hendelman W, Byszewski A. Formation of medical student professional identity: categorizing lapses of professionalism, and the learning environment. BMC Med Educ. 2014;14(1):Article139.

Sternszus R, Boudreau JD, Cruess RL, Cruess SR, Macdonald ME, Steinert Y. Clinical teachers’ perceptions of their role in professional identity formation. Acad Med. 2020;95(10):1594–9.

Jarvis-Selinger S, MacNeil KA, Costello GRL, Lee K, Holmes CL. Understanding professional identity formation in early clerkship: a novel framework. Acad Med. 2019;94(10):1574–80.

Sadeghi Avval Shahr H, Yazdani S, Afshar L. Professional socialization: an analytical definition. J Med Ethics Hist Med. 2019;12(17).

Brody H, Doukas D, Professionalism. A framework to guide medical education. Med Educ. 2014;48(10):980–7.

Cruess RL, Cruess SR, Boudreau JD, Snell L, Steinert Y. A schematic representation of the professional identity formation and socialization of medical students and residents: a guide for medical educators. Acad Med. 2015;90(6):718–25.

Irby DM, Hamstra SJ. Parting the clouds: three professionalism frameworks in medical education. Acad Med. 2016;91(12):1606–11.

Huda N, Faden L, Wilson CA, Plouffe RA, Li E, Saini MK et al. The ebb and flow of identity formation and competence development in sub-specialty residents: study of a continuity training setting. [Preprint]. 2020.

Remmen R, Denekens J, Scherpbier A, Hermann I, van der Vleuten C, Royen PV, et al. An evaluation study of the didactic quality of clerkships. Med Educ. 2000;34(6):460–4.

Wearne SM, Butler L, Jones JA. Educating registrars in your practice. Aust Fam Physician. 2016;45(5):274–7.

Balmer DF, Serwint JR, Ruzek SB, Giardino AP. Understanding paediatric resident-continuity preceptor relationships through the lens of apprenticeship learning. Med Educ. 2008;42(9):923–9.

Leeuw J-VD, Buwalda HGAR, Wieringa-De Waard N, Van Dijk M. Learning from a role model: a cascade or whirlpool effect? Med Teach. 2015;37(5):482–9.

Lightman E, Kingdon S, Nelson M. A prolonged assistantship for final-year students. Clin Teach. 2015;12(2):115–20.

Nirodi P, El-Sayeh H, Henfrey H. Applying the apprenticeship model to psychiatry: an evaluation. Prog Neurol Psychiatry. 2018;22(1):25–9.

Ratanawongsa N, Teherani A, Hauer KE. Third-year medical students’ experiences with dying patients during the internal medicine clerkship: a qualitative study of the informal curriculum. Acad Med. 2005;80(7):641–7.

Dréano-Hartz S, Rhondali W, Ledoux M, Ruer M, Berthiller J, Schott A-M, et al. Burnout among physicians in palliative care: impact of clinical settings. Palliat Support Care. 2016;14(4):402–10.

Koh MYH, Hum AYM, Khoo HS, Ho AHY, Chong PH, Ong WY, et al. Burnout and resilience after a decade in palliative care: what survivors have to teach us. A qualitative study of palliative care clinicians with more than 10 years of experience. J Pain Symptom Manage. 2020;59(1):105–15.

Lehto RH, Heeter C, Forman J, Shanafelt T, Kamal A, Miller P, et al. Hospice employees’ perceptions of their work environment: a focus group perspective. Int J Environ Res Public Health. 2020;17(17):6147.

Koh MYH, Chong PH, Neo PSH, Ong YJ, Yong WC, Ong WY, et al. Burnout, psychological morbidity and use of coping mechanisms among palliative care practitioners: a multi-centre cross-sectional study. Palliat Med. 2015;29(7):633–42.

Back AL, Steinhauser KE, Kamal AH, Jackson VA. Building resilience for palliative care clinicians: an approach to burnout prevention based on individual skills and workplace factors. J Pain Symptom Manage. 2016;52(2):284–91.

Kavalieratos D, Siconolfi DE, Steinhauser KE, Bull J, Arnold RM, Swetz KM, et al. “It is like heart failure. It is chronic...And it will kill you”: A qualitative analysis of burnout among hospice and palliative care clinicians. J Pain Symptom Manage 2017;53(5):901–910.e901.

Ercolani G, Varani S, Peghetti B, Franchini L, Malerba MB, Messana R, et al. Burnout in home palliative care: what is the role of coping strategies? J Palliat Care. 2020;35(1):46–52.

Tan BYQ, Kanneganti A, Lim LJH, Tan M, Chua YX, Tan L, et al. Burnout and associated factors among health care workers in Singapore during the COVID-19 pandemic. J Am Med Dir Assoc. 2020;21(12):1751–e17581755.

Dijxhoorn A-FQ, Brom L, van der Linden YM, Leget C, Raijmakers NJ. Prevalence of burnout in healthcare professionals providing palliative care and the effect of interventions to reduce symptoms: a systematic literature review. Palliat Med. 2021;35(1):6–26.

Teo YH, Peh TY, Abdurrahman A, Lee A, Chiam M, Fong W et al. A modified delphi approach to enhance nurturing of professionalism in postgraduate medical education in Singapore. Singap Med J. 2021.

Hee JM, Yap HW, Ong ZX, Quek SQM, Toh YP, Mason S, et al. Understanding the mentoring environment through thematic analysis of the learning environment in medical education: a systematic review. JJ Gen Intern Med. 2019;34(10):2190–9.

Lim SYS, Koh EYH, Tan BJX, Toh YP, Mason S, Krishna LKR. Enhancing geriatric oncology training through a combination of novice mentoring and peer and near-peer mentoring: a thematic analysis ofmentoring in medicine between 2000 and 2017. J Geriatr Oncol. 2020;11(4):566–75.

Kilbertus F, Ajjawi R, Archibald DB. You’re not trying to save somebody from death: learning as becoming in palliative care. Acad Med. 2018;93(6):929–36.

Burford B. Group processes in medical education: learning from social identity theory. Med Educ. 2012;46(2):143–52.

Sawatsky AP, Nordhues HC, Merry SP, Bashir MU, Hafferty FW. Transformative learning and professional identity formation during international health electives: a qualitative study using grounded theory. Acad Med. 2018;93(9):1381–90.

Soo J, Brett-MacLean P, Cave M-T, Oswald A. At the precipice: a prospective exploration of medical students’ expectations of the pre-clerkship to clerkship transition. Adv Health Sci Educ Theory Pract. 2016;21(1):141–62.

Hamstra SJ, Woodrow SI, Mangrulkar RS. Feeling pressure to stay late: Socialisation and professional identity formation in graduate medical education. Med Educ. 2008;42(1):7–9.

Wilson I, Cowin LS, Johnson M, Young H. Professional identity in medical students: pedagogical challenges to medical education. Teach Learn Med. 2013;25(4):369–73.

Wald HS, White J, Reis SP, Esquibel AY, Anthony D. Grappling with complexity: medical students’ reflective writings about challenging patient encounters as a window into professional identity formation. Med Teach. 2019;41(2):152–60.

Matthews R, Smith-Han K, Nicholson H. From physiotherapy to the army: negotiating previously developed professional identities in mature medical students. Adv Health Sci Educ Theory Pract. 2020;25(3):607–27.

Stuart E, O’Leary D, Rowntree R, Carey C, O’Rourke L, O’Brien E, et al. Challenges in experiential learning during transition to clinical practice: a comparative analysis of reflective writing assignments during general practice, paediatrics and psychiatry clerkships. Med Teach. 2020;42(11):1275–82.

Rosenthal S, Howard B, Schlussel YR, Herrigel D, Smolarz BG, Gable B, et al. Humanism at heart: preserving empathy in third-year medical students. Acad Med. 2011;86(3):350–8.

Wright SM, Levine RB, Beasley B, Haidet P, Gress TW, Caccamese S, et al. Personal growth and its correlates during residency training. Med Educ. 2006;40(8):737–45.

Levine RB, Haidet P, Kern DE, Beasley BW, Bensinger L, Brady DW, et al. Personal growth during internship. J Gen Intern Med. 2006;21(6):564–9.

Fischer MA, Haley H-L, Saarinen CL, Chretien KC. Comparison of blogged and written reflections in two medicine clerkships. Med Educ. 2011;45(2):166–75.

Kern DE, Wright SM, Carrese JA, Lipkin M Jr., Simmons JM, Novack DH, et al. Personal growth in medical faculty: a qualitative study. West J Med. 2001;175(2):92–8.

Kimmons R, Veletsianos G. The fragmented educator 2.0: Social networking sites, acceptable identity fragments, and the identity constellation. Comput Educ. 2014;72:292–301.

Gosselink MJ. Medical weblogs: advocacy for positive cyber role models. Clin Teach. 2011;8(4):245–8.

Fieseler C, Meckel M, Ranzini G. Professional personae - how organizational identification shapes online identity in the workplace. J Comput Mediat Commun. 2014;20(2):153–70.

Stokes J, Price B. Social media, visual culture and contemporary identity. Open Cybern Syst J. 2017:159–63.

Maghrabi RO, Oakley RL, Nemati HR. The impact of self-selected identity on productive or perverse social capital in social network sites. Comput Hum Behav. 2014;33:367–71.

Hojat M, Vergare MJ, Maxwell K, Brainard G, Herrine SK, Isenberg GA, et al. The devil is in the third year: a longitudinal study of erosion of empathy in medical school. Acad Med. 2009;84(9):1182–91.

Newton BW, Barber L, Clardy J, Cleveland E, O’Sullivan P. Is there hardening of the heart during medical school? Acad Med. 2008;83(3):244–9.

Kaczmarczyk JM, Chuang A, Dugoff L, Abbott JF, Cullimore AJ, Dalrymple J, et al. E-professionalism: a new frontier in medical education. Teach Learn Med. 2013;25(2):165–70.

Brown J, Reid H, Dornan T, Nestel D. Becoming a clinician: Trainee identity formation within the general practice supervisory relationship. Med Educ. 2020;54(11):993–1005.

Henschen BL, Bierman JA, Wayne DB, Ryan ER, Thomas JX, Curry RH, et al. Four-year educational and patient care outcomes of a team-based primary care longitudinal clerkship. Acad Med. 2015;90(11 Suppl):S43–49.

Lim-Dunham JE, Ensminger DC, McNulty JA, Hoyt AE, Chandrasekhar AJ. A vertically integrated online radiology curriculum developed as a cognitive apprenticeship: impact on student performance and learning. Acad Radiol. 2016;23(2):252–61.

Sen Gupta TK, Muray RB, McDonell A, Murphy B, Underhill AD. Rural internships for final year students: clinical experience, education and workforce. Rural Remote Health. 2008;8(1):827.

Yardley S, Teunissen PW, Dornan T. Experiential learning: AMEE guide 63. Med Teach. 2012;34(2):e102–115.

Dickinson BL, Gibson K, VanDerKolk K, Greene J, Rosu CA, Navedo DD et al. It is this very knowledge that makes us doctors: An applied thematic analysis of how medical students perceive the relevance of biomedical science knowledge to clinical medicine. BMC Med Educ. 2020;20(1):Article 356.

Sarraf-Yazdi S, Teo YN, How AEH, Teo YH, Goh S, Kow CS, et al. A scoping review of professional identity formation in undergraduate medical education. J Gen Intern Med. 2021;36(11):3511–21.

Alford CL, Currie DM. Introducing first-year medical students to clinical practice by having them shadow third-year clerks. Teach Learn Med. 2004;16(3):260–3.

Boudreau JD, Macdonald ME, Steinert Y. Affirming professional identities through an apprenticeship: insights from a four-year longitudinal case study. Acad Med. 2014;89(7):1038–45.

Peer KS. Professional identity formation: considerations for athletic training education. Athl Train Educ J. 2016;11(3):125–6.

Klamen DL, Williams R, Hingle S. Getting real: aligning the learning needs of clerkship students with the current clinical environment. Acad Med. 2019;94(1):53–8.

Régo P, Peterson R, Callaway L, Ward M, O’Brien C, Donald K. Using a structured clinical coaching program to improve clinical skills training and assessment, as well as teachers’ and students’ satisfaction. Med Teach. 2009;31(12):e586–595.

Gheasuddin AN, Misra R, Patel J. Use of an apprenticeship model to facilitate prescribing learning on clinical placements. Med Teach. 2022;44(8):940.

Bettin KA. The role of mentoring in the professional identity formation of medical students. Orthop Clin North Am. 2020;52(1).

Baerheim A, Thesen J. Medical students’ evaluation of preceptorship in general practice in Vestlandet. Tidsskr nor Laegeforen. 2003;123(16):2271–3.

Golden BP, Henschen BL, Gard LA, Ryan ER, Evans DB, Bierman J, et al. Learning to be a doctor: medical students’ perception of their roles in longitudinal outpatient clerkships. Patient Educ Couns. 2018;101(11):2018–24.

Harris GD, Professionalism. Part ii - teaching and assessing the learner’s professionalism. Fam Med. 2004;36(6):390–2.

Sheu L, Goglin S, Collins S, Cornett P, Clemons S, O’Sullivan PS. How do clinical electives during the clerkship year influence career exploration? A qualitative study. Teach Learn Med. 2021:1–11.

Hay A, Smithson S, Mann K, Dornan T. Medical students’ reactions to an experience-based learning model of clinical education. Perspect Med Educ. 2013;2(2):58–71.

Abbey L, Willett R, Selby-Penczak R, McKnight R. Social learning: medical student perceptions of geriatric house calls. Gerontol Geriatr Educ. 2010;31(2):149–62.

Côté L, Laughrea PA. Preceptors’ understanding and use of role modeling to develop the canmeds competencies in residents. Acad Med. 2014;89(6):934–9.

Goldstein EA, MacLaren CF, Smith S, Mengert TJ, Maestas RR, Foy HM, et al. Promoting fundamental clinical skills: a competency-based college approach at the University of Washington. Acad Med. 2005;80(5):423–33.

Jones K, Reis S. Learning through vulnerability: a mentor-mentee experience. Ann Fam Med. 2010;8(6):552–5.

Kalén S, Stenfors-Hayes T, Hylin U, Larm MF, Hindbeck H, Ponzer S. Mentoring medical students during clinical courses: a way to enhance professional development. Med Teach. 2010;32(8):e315–321.

Tariq M, Iqbal S, Haider SI, Abbas A. Using the cognitive apprenticeship model to identify learning strategies that learners view as effective in ward rounds. Postgrad Med J. 2021;97(1143):5–9.

Braniff C, Spence RA, Stevenson M, Boohan M, Watson P. Assistantship improves medical students’ perception of their preparedness for starting work. Med Teach. 2016;38(1):51–8.

Iwata K, Gill D. Learning through work: clinical shadowing of junior doctors by first year medical students. Med Teach. 2013;35(8):633–8.

Bleakley A. Pre-registration house officers and ward-based learning: a ‘new apprenticeship’ model. Med Educ. 2002;36(1):9–15.

Brody DS, Ryan K, Kuzma MA, Suppl. S105–109.

Sandelowski M, Barroso J. Handbook for synthesizing qualitative research. Springer Publishing Company; 2006.

Ng YX, Koh ZYK, Yap HW, Tay KT, Tan XH, Ong YT, et al. Assessing mentoring: a scoping review of mentoring assessment tools in internal medicine between 1990 and 2019. PLoS ONE. 2020;15(5):e0232511.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc. 2015;13(3):141–6.

Sambunjak D, Straus SE, Marusic A. A systematic review of qualitative research on the meaning and characteristics of mentoring in academic medicine. J Gen Intern Med. 2010;25(1):72–8.

Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M et al. Guidance on the conduct of narrative synthesis in systematic reviews. J Epidemiol Community Health. 2006;59(1).

Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. Rameses publication standards: Meta-narrative reviews. BMC Med. 2013;11(1):Article20.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

Ong YT, Quek CWN, Pisupati A, Loh EKY, Venktaramana V, Chiam M, et al. Mentoring future mentors in undergraduate medical education. PLoS ONE. 2022;17(9):e0273358–0273358.

Venktaramana V, Ong YT, Yeo JW, Pisupati A, Krishna LKR. Understanding mentoring relationships between mentees, peer and senior mentors. BMC Med Educ. 2023;23(1):76.

Chan JSK, Lau D, King E, Roever L, Liu T, Shum Y, et al. Virtual medical research mentoring. Clin Teach. 2023;20(4):e13598.

Joe MB, Cusano A, Leckie J, Czuczman N, Exner K, Yong H, et al. Mentorship programs in residency: a scoping review. J Grad Med Educ. 2023;15(2):190–200.

Lupi A, Shu L, Lopez A. Local mentoring as a strategy to recruit a more diverse physician workforce. Acad Med. 2022;97(6):779–80.

Krishna LKR, Pisupati A, Ong YT, Teo KJH, Teo MYK, Venktaramana V, et al. Assessing the effects of a mentoring program on professional identity formation. BMC Med Educ. 2023;23(1):Article799.

Krishna LKR, Pisupati A, Teo KJH, Teo MYK, Quek CWN, Chua KZY, et al. Professional identity formation amongst peer-mentors in a research-based mentoring programme. BMC Med Educ. 2023;23(1):Article787.

Munoz JA, Sarmiento MA, Alejandra Esparza Y, Shipp A, Luisa Veloz A, Edith Esparza Y, et al. Mentoring medical students: voices from Zoom rooms during the pandemic. Int J Divers Educ. 2022;23(1):25–34.

Drossard S, Härtl A. Development and implementation of digital peer mentoring in small groups for first-year medical students. GMS J Med Educ. 2024;41(1):Doc11.

Scholz A, Gehres V, Schrimpf A, Bleckwenn M, Deutsch T, Geier AK. Long-term mentoring relationships in undergraduate longitudinal general practice tracks - a qualitative study on the perspective of students and general practitioners. Med Educ Online. 2023;28(1):2149252.

Shen MR, Tzioumis E, Andersen E, Wouk K, McCall R, Li W, et al. Impact of mentoring on academic career success for women in medicine: a systematic review. Acad Med. 2022;97(3):444–58.

Hartlage CS, Sosa DM. Everyone deserves a seat at the table: mentoring to uplift minoritized trainees. Acad Med. 2023.

Lee SI, Bluemke DA. Mentoring in academic radiology. Radiology. 2022;303(1):E20–2.

Sharma A, Leeper H, Bang S, Molaie D, Porter AB. The impact of mentoring on early career faculty: Assessment of a virtual mentoring program. J Clin Oncol. 2022;40(16).

Murphy M, Record H, Callander JK, Dohan D, Grandis JR. Mentoring relationships and gender inequities in academic medicine: findings from a multi-institutional qualitative study. Acad Med. 2022;97(1):136–42.

Farrukh K, Hoor T. Online mentoring session during COVID-19: experiences of mentees and mentors - a phenomenology. Prof Med J. 2022;29(12):1886–91.

Bui DT, Barnett T, Hoang H, Chinthammit W. Development of a framework to support situational tele-mentorship of rural and remote practice. Med Teach. 2023;45(6):642–9.

Ramani S, Chugh N, Chisolm MS, Hays R, McKimm J, Kusurkar R, et al. Mentoring relationships: a mentee’s journey. Acad Med. 2023;98(3):423.

Williams JS, Walker RJ, Burgess KM, Shay LA, Schmidt S, Tsevat J, et al. Mentoring strategies to support diversity in research-focused junior faculty: a scoping review. J Clin Transl Sci. 2023;7(1):e21.

Koven S. What is a mentor? N Engl J Med. 2024;390(8):683–5.

Jafari M, Moodi Ghalibaf A. Peer-research learning and mentoring for undergraduate medical students: benefits and challenges. Res Dev Med Educ. 2022;11:19.

Kalet A, Libby AM, Jagsi R, Brady K, Chavis-Keeling D, Pillinger MH, et al. Mentoring underrepresented minority physician-scientists to success. Acad Med. 2022;97(4):497–502.

Koh EYH, Koh KK, Renganathan Y, Krishna L. Role modelling in professional identity formation: a systematic scoping review. BMC Med Educ. 2023;23(1):194.

Stadtlander L, Ozcan A, Johnson L, Nicholson B, Hyder N. Faculty and student online mentoring preferences. J Educ Res Pract. 2022;12(1).

Heffernan L, O’Dowd E. Em and me: near-peer mentoring in an emergency department. Ir Med J. 2023;116(10):874–83.

Behkam S, Tavallaei A, Maghbouli N, Mafinejad MK, Ali JH. Students’ perception of educational environment based on Dundee ready education environment measure and the role of peer mentoring: a cross-sectional study. BMC Med Educ. 2022;22(1):Article176.

Pfund C, Sancheznieto F, Byars-Winston A, Zárate S, Black S, Birren B et al. Evaluation of a culturally responsive mentorship education program for the advisers of howard hughes medical institute gilliam program graduate students. CBE Life Sci Educ. 2022;21(3):Article 50.

Hill SEM, Ward WL, Seay A, Buzenski J. The nature and evolution of the mentoring relationship in academic health centers. J Clin Psychol Med Settings. 2022;29(3):557–69.

Jan S, Mahboob U. Online mentoring: challenges and strategies. Pak J Med Sci. 2022;38(8):2272–7.

Krishna L, Toh YP, Mason S, Kanesvaran R. Mentoring stages: a study of undergraduate mentoring in palliative medicine in Singapore. PLoS ONE. 2019;14(4):e0214643.

Hee J, Toh YL, Yap HW, Toh YP, Kanesvaran R, Mason S, et al. The development and design of a framework to match mentees and mentors through a systematic review and thematic analysis of mentoring programs between 2000 and 2015. Mentor Tutoring: Partnersh Learn. 2020;28(3):340–64.

Schrempf S, Herrigel L, Pohlmann J, Griewatz J, Lammerding-Köppel M. Everybody is able to reflect, or aren’t they? Evaluating the development of medical professionalism via a longitudinal portfolio mentoring program from a student perspective. GMS J Med Educ. 2022;39(1):Doc12.

Junn JC, Whitman GJ, Wasnik AP, Wang MX, Guelfguat M, Goodman ED, et al. Virtual mentoring: a guide to navigating a new age in mentorship. Acad Radiol. 2023;30(4):749–54.

Lakhani DA, Swaney KJ, Hogg JP. Resident managed peer-mentoring program: a novel way to engage medical students and radiology residents in collaborative research. Acad Radiol. 2022;29(9):1425–31.

Dancer JM. Mentoring in healthcare: theory in search of practice? Clin Manag. 2003;12(1):21–31.

Welsh ET, Bhave D, yong Kim K. Are you my mentor? Informal mentoring mutual identification. Career Dev Int. 2012;17(2):137–48.

Reiss TF, Moss J, Watkins TR, Malhotra A. BEAR cage: mentoring through engagement. Am J Respir Crit Care Med. 2016;193(7):714–6.

Sonawane T, Meshram R, Jagia G, Gajbhiye R, Adhikari S. Effects of mentoring in first year medical undergraduate students using DASS-21. J Clin Diagn Res. 2021;15(11):7–10.

Stamm M, Buddeberg-Fischer B. The impact of mentoring during postgraduate training on doctors’ career success. Med Educ. 2011;45(5):488–96.

Yehia BR, Cronholm PF, Wilson N, Palmer SC, Sisson SD, Guilliames CE et al. Mentorship and pursuit of academic medicine careers: A mixed methods study of residents from diverse backgrounds. BMC Med Educ. 2014;14(1):Article 26.

Eby LT, Butts MM, Durley J, Ragins BR. Are bad experiences stronger than good ones in mentoring relationships? Evidence from the protégé and mentor perspective. J Vocat Behav. 2010;77(1):81–92.

Mohd Shafiaai MSF, Kadirvelu A, Pamidi N. Peer mentoring experience on becoming a good doctor: student perspectives. BMC Med Educ. 2020;20(1):Article494.

Morrison LJ, Lorens E, Bandiera G, Liles WC, Lee L, Hyland R, et al. Impact of a formal mentoring program on academic promotion of department of medicine faculty: a comparative study. Med Teach. 2014;36(7):608–14.

Ng CH, Ong ZH, Koh JWH, Ang RZE, Tan LHS, Tay KT, et al. Enhancing interprofessional communications training in internal medicine. Lessons drawn from a systematic scoping review from 2000 to 2018. J Contin Educ Health Prof. 2020;40(1):27–35.

Pinilla S, Pander T, von der Borch P, Fischer MR, Dimitriadis K. 5 years of experience with a large-scale mentoring program for medical students. GMS Z Med Ausbild. 2015;32(1):Doc5.

Sng JH, Pei Y, Toh YP, Peh TY, Neo SH, Krishna LKR. Mentoring relationships between senior physicians and junior doctors and/or medical students: a thematic review. Med Teach. 2017;39(8):866–75.

Farid H, Bain P, Huang G. A scoping review of peer mentoring in medicine. Clin Teach. 2022;19(5):e13512.

Li MK, Adus SL, Weyman K. There’s always something to talk about! The unexpected benefits of going virtual in a Canadian diversity mentorship program. Can Med Educ J. 2022;13(1):90–2.

Frizell CA, Caruthers KL, Sturges D. Intentional mentoring of healthcare provider students from underrepresented groups in medicine. Med Sci Educ. 2023;33(3):807–8.

Aziz A, Shadab W, Siddique L, Mahboob U. Exploring the experiences of struggling undergraduate medical students with formal mentoring program at a private medical college in Rawalpindi. Pak J Med Sci. 2023;39(3):815–9.

Penaloza NG, Ardines KEZ, Does S, Washington SL III, Tandel MD, Braddock CH III, et al. Someone like me: an examination of the importance of race-concordant mentorship in urology. Urology. 2023;171:41–8.

Silver JK. Six practical strategies to mentor and sponsor women in academic medicine. J Med Internet Res. 2023;25:e47799.

Verhoef L, Vivekanantham A, Berti A, Bolek E, Smeele H, Oztas M, et al. The emerging eular network (emeunet) peer-review mentoring program: ten years of initiative. Ann Rheum Dis. 2022;81:202–3.

Chia EWY, Tay KT, Xiao S, Teo YH, Ong YT, Chiam M, et al. The pivotal role of host organizations in enhancing mentoring in internal medicine: a scoping review. J Med Educ Curric Dev. 2020;7:2382120520956647.

Cameron KA, Rodrigues TJ, Keswani RN. Developing a mentoring network to navigate fellowship and beyond: moving from mentor to mentors. Gastroenterology. 2024;166(5):723–e727721.

Silver JK, Gavini N. The push-pull mentoring model: ensuring the success of mentors and mentees. J Med Internet Res. 2023;25:e48037.

Kuzman M. Academic mentoring for psychiatric trainees during the pandemic. Eur Psychiatry. 2022;65:S60.

Tetzlaff J, Lomberk G, Smith HM, Agrawal H, Siegel DH, Apps JN. Adapting mentoring in times of crisis: what we learned from COVID-19. Acad Psychiatry. 2022;46(6):774–9.

Barron B. Interest and self-sustained learning as catalysts of development: a learning ecology perspective. Hum Dev. 2006;49(4):193–224.

Karcher MJ, Kuperminc GP, Portwood SG, Sipe CL, Taylor AS. Mentoring programs: a framework to inform program development, research, and evaluation. J Community Psychol. 2006;34(6):709–25.

Wells JN, Cagle CS. Preparation and participation of undergraduate students to inform culturally sensitive research. Nurse Educ Today. 2009;29(5):505–9.

Bozeman B, Feeney MK. Toward a useful theory of mentoring: a conceptual analysis and critique. Admin Soc. 2007;39(6):719–39.

Indyk D, Deen D, Fornari A, Santos MT, Lu W-H, Rucker L. The influence of longitudinal mentoring on medical student selection of primary care residencies. BMC Med Educ. 2011;11(1):Article27.

Meeuwissen SN, Stalmeijer RE, Govaerts M. Multiple-role mentoring: mentors’ conceptualisations, enactments and role conflicts. Med Educ. 2019;53(6):605–15.

Rangachari D, Brown LE, Kern DE, Melia MT. Clinical coaching: evolving the apprenticeship model for modern housestaff. Med Teach. 2017;39(7):780–2.

Pront L, Gillham D, Schuwirth LW. Competencies to enable learning-focused clinical supervision: a thematic analysis of the literature. Med Educ. 2016;50(4):485–95.

Yang MM, Golden BP, Cameron KA, Gard L, Bierman JA, Evans DB, et al. Learning through teaching: peer teaching and mentoring experiences among third-year medical students. Teach Learn Med. 2022;34(4):360–7.

Lynn L, Stadtlander L, Inman D, Burkholder G, Morgan A. Transforming doctoral mentoring expectations and culture: an action model approach. New Dir Teach Learn. 2023;2023(176):75–82.

Moore JL, Myers A, McConnell H. Mentoring high-impact undergraduate research experiences. Pedagogy. 2022;22(1):17–21.

Mishra MK. Evidence-based learning modules and culturally responsive mentoring to engage underserved undergraduate students. J Immunol. 2022;208(1Supplement):106116.

Moawad H. Best practices for mentoring new physicians. Oncology. 2022 2022/02//:127.

Hoffmeyer C, Milliren A, Eckstein D. The Hoffmeyer mentoring activity checklist: invitations to professional growth. J Invitational Theory Pract. 2022;11:54–62.

Palmeri M, Bono K, Huang A, Gunther JR, Mattes MD. Characterization of research mentorship during medical school for future radiation oncology trainees. Adv Radiat Oncol. 2024;9(5):101460.

De Rosa S, Battaglini D, Bennett V, Rodriguez-Ruiz E, Zaher AMS, Galarza L, et al. Key steps and suggestions for a promising approach to a critical care mentoring program. J Anesth Analg Crit Care. 2023;3(1):30.

Jacob RA, Williams PN, Chisty A. Twelve tips for developing and maintaining a successful peer mentoring program for junior faculty in academic medicine. Med Teach. 2022;45(3):252–6.

Templeton NR, Jeong S, Villarreal E. Editorial overview: mentoring to support professional knowledge. Mentor Tutoring: Partnersh Learn. 2022;30(3):275–9.

Rallis KS, Wozniak A, Hui S, Stammer A, Cinar C, Sun M, et al. Mentoring medical students towards oncology: results from a pilot multi-institutional mentorship programme. J Cancer Educ. 2022;37(4):1053–65.

Kusner JJ, Chen JJ, Saldaña F, Potter J. Aligning student-faculty mentorship expectations and needs to promote professional identity formation in undergraduate medical education. J Med Educ Curric Dev. 2022;9:23821205221096307.

Khatun M, Akter P, Yunus S, Alam K, Pedersen C, Byrskog U, et al. Challenges to implement evidence-based midwifery care in Bangladesh. An interview study with medical doctors mentoring health care providers. Sex Reprod Healthc. 2022;31:100692.

Torre DM, Daley BJ, Sebastian JL, Elnicki DM. Overview of current learning theories for medical educators. Am J Med. 2006;119(10):903–7.

Gormley B. An application of attachment theory: mentoring relationship dynamics and ethical concerns. Mentor Tutoring: Partnersh Learn. 2008;16(1):45–62.

Taylor DC, Hamdy H. Adult learning theories: implications for learning and teaching in medical education: AMEE guide 83. Med Teach. 2013;35(11):e1561–72.

Lewis C, Olshansky E. Relational-cultural theory as a framework for mentoring in academia: toward diversity and growth-fostering collaborative scholarly relationships. Mentor Tutoring: Partnersh Learn. 2016;24(5):383–98.

Heeneman S, de Grave W. Tensions in mentoring medical students toward self-directed and reflective learning in a longitudinal portfolio-based mentoring system–an activity theory analysis. Med Teach. 2017;39(4):368–76.

Irby BJ, Boswell JN, Searby LJ, Kochan F, Garza R, Abdelrahman N. The Wiley international handbook of mentoring: Wiley; 2020.

Wadi MM, Yusoff MSB, Taha MH, Shorbagi S, Nik Lah NAZ, Abdul Rahim AF. The framework of systematic Assessment for Resilience (SAR): development and validation. BMC Med Educ. 2023;23(1): Article 213.

Download references

Acknowledgements

The authors would like to thank Annelissa Chew Chin for her expert guidance and advice in designing our search strategy. The authors would like to dedicate this paper to the late Dr. S Radha Krishna and A/Prof Cynthia Goh whose advice and ideas were integral to the success of this review and Thondy, Maia Olivia and Raja Kamarul whose lives continue to inspire us.The authors would also like to thank the anonymous reviewers for their helpful comments which greatly enhanced this manuscript.

No funding was received for this review.

Author information

Authors and affiliations.

Yong Loo Lin School of Medicine, National University of Singapore, Level 11 NUHS Tower Block, 1E Kent Ridge Road, Singapore, 119228, Singapore

Mac Yu Kai Teo, Casper Keegan Ronggui Lin, Ranitha Govindasamy, Jia Ling Goh, Yi Zhou, Kuang Teck Tay, Ryan Rui Song Ong, Vanessa Tan, Youru Toh, Anushka Pisupati, Vijayprasanth Raveendran, Keith Zi Yuan Chua, Elaine Li Ying Quah, Jeevasuba Sivakumar, Samyuktha Dhanalakshmi Senthilkumar, Keerthana Suresh, Wesley Teck Wee Loo, Ruth Si Man Wong, Yiying Pei, Julia Huina Sng, Simone Qian Min Quek & Lalit Kumar Radha Krishna

Division of Supportive and Palliative Care, National Cancer Centre Singapore, 30 Hospital Boulevard, Singapore, 168583, Singapore

Mac Yu Kai Teo, Casper Keegan Ronggui Lin, Jia Ling Goh, Yi Zhou, Kuang Teck Tay, Ryan Rui Song Ong, Vanessa Tan, Youru Toh, Anushka Pisupati, Vijayprasanth Raveendran, Keith Zi Yuan Chua, Elaine Li Ying Quah, Jeevasuba Sivakumar, Samyuktha Dhanalakshmi Senthilkumar, Keerthana Suresh, Wesley Teck Wee Loo, Ruth Si Man Wong, Yiying Pei, Julia Huina Sng, Simone Qian Min Quek, Eng Koon Ong, Gillian Li Gek Phua & Lalit Kumar Radha Krishna

Division of Cancer Education, National Cancer Centre Singapore, 30 Hospital Boulevard, Singapore, 168583, Singapore

Mac Yu Kai Teo, Casper Keegan Ronggui Lin, Nur Amira Binte Abdul Hamid, Ranitha Govindasamy, Nagavalli Somasundaram, Jia Ling Goh, Yi Zhou, Kuang Teck Tay, Ryan Rui Song Ong, Vanessa Tan, Youru Toh, Anushka Pisupati, Vijayprasanth Raveendran, Keith Zi Yuan Chua, Elaine Li Ying Quah, Jeevasuba Sivakumar, Samyuktha Dhanalakshmi Senthilkumar, Keerthana Suresh, Wesley Teck Wee Loo, Ruth Si Man Wong, Yiying Pei, Julia Huina Sng, Simone Qian Min Quek, Jasmine Lerk Juan Owyong, Ting Ting Yeoh, Eng Koon Ong, Simon Yew Kuang Ong & Lalit Kumar Radha Krishna

Department of Medical Sciences, Khalifa University College of Medicine and Health Sciences, Abu Dhabi, United Arab Emirates

Halah Ibrahim

Department of Pharmacy, National Cancer Centre Singapore, 30 Hospital Boulevard, Singapore, 168583, Singapore

Casper Keegan Ronggui Lin & Ting Ting Yeoh

Centre for Biomedical Ethics, National University of Singapore, Blk MD11, 10 Medical Drive, Singapore, #02-03, 117597, Singapore

Casper Keegan Ronggui Lin, Ranitha Govindasamy & Lalit Kumar Radha Krishna

Duke-NUS Medical School, National University of Singapore, 8 College Road, Singapore, 169857, Singapore

Nagavalli Somasundaram, Eng Koon Ong, Anupama Roy Chowdhury, Simon Yew Kuang Ong & Lalit Kumar Radha Krishna

Division of Medical Oncology, National Cancer Centre Singapore, 30 Hospital Boulevard, Singapore, 168583, Singapore

Nagavalli Somasundaram, Crystal Lim & Simon Yew Kuang Ong

Medical Social Services, Singapore General Hospital, Block 3, Singapore, 169854, Singapore

Crystal Lim

Assisi Hospice, 832 Thomson Road, Singapore, 574627, Singapore

Eng Koon Ong

Lien Centre for Palliative Care, Duke-NUS Medical School, National University of Singapore, 8 College Road, Singapore, 169857, Singapore

Gillian Li Gek Phua

Palliative Care Institute Liverpool, Academic Palliative & End of Life Care Centre, Cancer Research Centre, University of Liverpool, 200 London Road, Liverpool, L3 9TA, UK

Stephen Mason & Lalit Kumar Radha Krishna

Health Data Science, University of Liverpool, Whelan Building The Quadrangle, Liverpool, Brownlow Hill, Liverpool, L69 3GB, UK

Ruaraidh Hill & Lalit Kumar Radha Krishna

Department of Geriatric Medicine, Singapore General Hospital, Academia, Level 3, College Road, Singapore, 169608, Singapore

Anupama Roy Chowdhury

PalC, The Palliative Care Centre for Excellence in Research and Education, PalC c/o Dover Park Hospice, 10 Jalan Tan Tock Seng, Singapore, 308436, Singapore

Lalit Kumar Radha Krishna

You can also search for this author in PubMed   Google Scholar

Contributions

All authors MYKT, HI, CKLR, NABAH, RG, NS, CL, JLG, YZ, KTT, RRSO, VT, TY, AP, VR, KZYC, ELYQ, JS, SDS, KS, WTWL, RSMW, YP, JHS, SQMQ, JLJO, TTY, EKO, GLGP, SM, RH, ARC, SYKO, and LKRK were involved in data curation, formal analysis, investigation, preparing the original draft of the manuscript as well as reviewing and editing the manuscript. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Lalit Kumar Radha Krishna .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Teo, M.Y.K., Ibrahim, H., Lin, C.K.R. et al. Mentoring as a complex adaptive system – a systematic scoping review of prevailing mentoring theories in medical education. BMC Med Educ 24 , 726 (2024). https://doi.org/10.1186/s12909-024-05707-5

Download citation

Received : 22 February 2024

Accepted : 25 June 2024

Published : 05 July 2024

DOI : https://doi.org/10.1186/s12909-024-05707-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical education
• Host organization
• Communities of practice
• Mentorship theories

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]