database individually, including ALL your search terms, any
MeSH or other subject headings, truncation (like hemipleg ),
and/or wildcards (like sul ur). Apply all your limits (such as
years of search, English language only, and so on). Once all
search terms have been combined and you have applied all
relevant limits, you should have a final number of records or
articles for each database. Enter this information in the top
left box of the PRISMA flow chart. You should add the total
number of combined results from all databases (including
duplicates) after the equal sign where it says .
Many researchers also add notations in the box for the number
of results from each database search, for example, Pubmed
(n=335), Embase (n= 600), and so on. If you search trial
registers, such as , , , or others,
you should enter that number after the equal sign in .
NOTE:Some citation managers automatically remove duplicates
with each file you import. Be sure to capture the number of articles
from your database searches before any duplicates are removed.
To avoid reviewing duplicate articles,
you need to remove any articles that appear more than once in your
results. You may want to export the entire list of articles from each
database to a citation manager such as EndNote, Sciwheel, Zotero,
or Mendeley (including both citation and abstract in your file) and
remove the duplicates there. If you are using Covidence for your
review, you should also add the duplicate articles identified in
Covidence to the citation manager number. Enter the number of
records removed as duplicates in the second box on your PRISMA
template. If you are using automation tools to help evaluate the
relevance of citations in your results, you would also enter that
number here.
If you are using Covidence to screen your articles, you can
copy the numbers from the PRISMA diagram in your Covidence
review into the boxes mentioned below. Covidence does not include
the number of results from each database, so you will need to keep
track of that number yourself.
The final step is to subtract the number
of records excluded during the review of full-texts (Step 9)
from the total number of full-texts reviewed (Step 8). Enter
this number in the box labeled "Studies included in review,"
combining numbers with your grey literature search results in this
box if needed.
You have now completed your PRISMA flow diagram, unless you
have also performed searches in non-database sources or are
performing a search update. If so, complete those portions of the template as well.
Step 1: Preparation Download the flow diagram template version 1 PRISMA 2020 flow diagram for new systematic reviews which included searches of databases, registers and other sources or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases, registers and other sources .
If you have identified articles through other sources than databases (such as manual searches through reference lists of articles you have found or search engines like Google Scholar), enter the total number of records from each source type in the box on the top right of the flow diagram. | |
This should be the total number of reports you obtain from each grey literature source. | |
List the number of documents for which you are unable to find the full text. Remember to use Find@UNC and to request items to see if we can order them from other libraries before automatically excluding them. | |
This should be the number of grey literature reports sought for retrieval (Step 2) minus the number of reports not retrieved (Step 3). Review the full text for these items to assess their eligibility for inclusion in your systematic review. | |
After reviewing all items in the full-text screening stage for eligibility, enter the total number of articles you exclude in the box titled "Reports Excluded," and then list your reasons for excluding the item as well as the number of items excluded for each reason. Examples include wrong setting, wrong patient population, wrong intervention, wrong dosage, etc. You should only count an excluded item once in your list even if if meets multiple exclusion criteria. | |
The final step is to subtract the number of excluded articles or records during the eligibility review of full-texts from the total number of articles reviewed for eligibility. Enter this number in the box labeled "Studies included in review," combining numbers with your database search results in this box if needed. You have now completed your PRISMA flow diagram, which you can now include in the results section of your article or assignment. |
Step 1: Preparation Download the flow diagram template version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases and registers only or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases, registers and other sources .
In the Previous
| |
At the bottom of the column, There will also be a box for the total number of studies included in your |
For more information about updating your systematic review, see the box Updating Your Review? on the Step 3: Conduct Literature Searches page of the guide.
Scientific articles often follow the IMRaD format: Introduction, Methods, Results, and Discussion. You will also need a title and an abstract to summarize your research.
You can read more about scientific writing through the library guides below.
Systematic reviews follow the same structure as original research articles, but you will need to report on your search instead of on details like the participants or sampling. Sections of your manuscript are shown as bold headings in the PRISMA checklist.
Title | Describe your manuscript and state whether it is a systematic review, meta-analysis, or both. |
---|---|
Abstract | Structure the abstract and include (as applicable): background, objectives, data sources, study eligibility criteria, participants, interventions, quality assessment and synthesis methods, results, limitations, conclusions, implications of key findings, and systematic review registration number. |
Introduction | Describe the rationale for the review and provide a statement of questions being addressed. |
Methods | Include details regarding the protocol, eligibility criteria, databases searched, full search strategy of at least one database (often reported in appendix), and the study selection process. Describe how data were extracted and analyzed. If a librarian is part of your research team, that person may be best suited to write this section. |
Results | Report the numbers of articles screened at each stage using a PRISMA diagram. Include information about included study characteristics, risk of bias (quality assessment) within studies, and results across studies. |
Discussion | Summarize main findings, including the strength of evidence and limitations of the review. Provide a general interpretation of the results and implications for future research. |
Funding | Describe any sources of funding for the systematic review. |
Appendix | Include entire search strategy for at least one database in the appendix (include search strategies for all databases searched for more transparency). |
Refer to the PRISMA checklist for more information.
Consider including a Plain Language Summary (PLS) when you publish your systematic review. Like an abstract, a PLS gives an overview of your study, but is specifically written and formatted to be easy for non-experts to understand.
Tips for writing a PLS:
Learn more about Plain Language Summaries:
Image: David Parmenter's Shop
• PRISMA Flow Diagram - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.
• PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis
PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram
Image by | from the UMB HSHSL Guide. (26 min) on how to conduct and write a systematic review from RMIT University from the VU Amsterdam . , (1), 6–23. https://doi.org/10.3102/0034654319854352 . (1), 49-60. . (4), 471-475. (2020) (2020) - Methods guide for effectiveness and comparative effectiveness reviews (2017) - Finding what works in health care: Standards for systematic reviews (2011) - Systematic reviews: CRD’s guidance for undertaking reviews in health care (2008) |
| entify your research question. Formulate a clear, well-defined research question of appropriate scope. Define your terminology. Find existing reviews on your topic to inform the development of your research question, identify gaps, and confirm that you are not duplicating the efforts of previous reviews. Consider using a framework like or to define you question scope. Use to record search terms under each concept. It is a good idea to register your protocol in a publicly accessible way. This will help avoid other people completing a review on your topic. Similarly, before you start doing a systematic review, it's worth checking the different registries that nobody else has already registered a protocol on the same topic. - Systematic reviews of health care and clinical interventions - Systematic reviews of the effects of social interventions (Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies) - The protocol is published immediately and subjected to open peer review. When two reviewers approve it, the paper is sent to Medline, Embase and other databases for indexing. - upload a protocol for your scoping review - Systematic reviews of healthcare practices to assist in the improvement of healthcare outcomes globally - Registry of a protocol on OSF creates a frozen, time-stamped record of the protocol, thus ensuring a level of transparency and accountability for the research. There are no limits to the types of protocols that can be hosted on OSF. - International prospective register of systematic reviews. This is the primary database for registering systematic review protocols and searching for published protocols. . PROSPERO accepts protocols from all disciplines (e.g., psychology, nutrition) with the stipulation that they must include health-related outcomes. - Similar to PROSPERO. Based in the UK, fee-based service, quick turnaround time. - Submit a pre-print, or a protocol for a scoping review. - Share your search strategy and research protocol. No limit on the format, size, access restrictions or license.outlining the details and documentation necessary for conducting a systematic review: , (1), 28. |
Clearly state the criteria you will use to determine whether or not a study will be included in your search. Consider study populations, study design, intervention types, comparison groups, measured outcomes. Use some database-supplied limits such as language, dates, humans, female/male, age groups, and publication/study types (randomized controlled trials, etc.). | |
Run your searches in the to your topic. Work with to help you design comprehensive search strategies across a variety of databases. Approach the grey literature methodically and purposefully. Collect ALL of the retrieved records from each search into , such as , or , and prior to screening. using the and . | |
- export your Endnote results in this screening software | Start with a title/abstract screening to remove studies that are clearly not related to your topic. Use your to screen the full-text of studies. It is highly recommended that two independent reviewers screen all studies, resolving areas of disagreement by consensus. |
Use , or systematic review software (e.g. , ), to extract all relevant data from each included study. It is recommended that you pilot your data extraction tool, to determine if other fields should be included or existing fields clarified. | |
Risk of Bias (Quality) Assessment - (download the Excel spreadsheet to see all data) | Use a Risk of Bias tool (such as the ) to assess the potential biases of studies in regards to study design and other factors. Read the to learn about the topic of assessing risk of bias in included studies. You can adapt ( ) to best meet the needs of your review, depending on the types of studies included. |
- - - | Clearly present your findings, including detailed methodology (such as search strategies used, selection criteria, etc.) such that your review can be easily updated in the future with new research findings. Perform a meta-analysis, if the studies allow. Provide recommendations for practice and policy-making if sufficient, high quality evidence exists, or future directions for research to fill existing gaps in knowledge or to strengthen the body of evidence. For more information, see: . (2), 217–226. https://doi.org/10.2450/2012.0247-12 - Get some inspiration and find some terms and phrases for writing your manuscript - Automated high-quality spelling, grammar and rephrasing corrections using artificial intelligence (AI) to improve the flow of your writing. Free and subscription plans available. |
- - | 8. Find the best journal to publish your work. Identifying the best journal to submit your research to can be a difficult process. To help you make the choice of where to submit, simply insert your title and abstract in any of the listed under the tab. |
Adapted from A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library
This diagram illustrates in a visual way and in plain language what review authors actually do in the process of undertaking a systematic review. |
This diagram illustrates what is actually in a published systematic review and gives examples from the relevant parts of a systematic review housed online on The Cochrane Library. It will help you to read or navigate a systematic review. |
Source: Cochrane Consumers and Communications (infographics are free to use and licensed under Creative Commons )
Check the following visual resources titled " What Are Systematic Reviews?"
Image: | - the methods of the systematic review are generally decided before conducting it.
Source: Foster, M. (2018). Systematic reviews service: Introduction to systematic reviews. Retrieved September 18, 2018, from |
A Systematic Literature Review (SLR) template is a structured framework used for conducting and documenting a systematic review of existing research studies on a specific topic or research question. Systematic literature reviews are commonly used in academic and research settings to provide a comprehensive and unbiased summary of the available literature on a particular subject. Here's a template for conducting a systematic literature review:
Systematic Literature Review (SLR) Template
Provide a clear and descriptive title for your systematic literature review.
2. Objective:
State the main research question or objectives of the systematic literature review.
3. Inclusion and Exclusion Criteria:
Define the criteria for selecting and excluding studies. This may include criteria related to publication date, study design, geographic location, and relevance to the research question.
4. Search Strategy:
Describe the search strategy used to identify relevant studies, including databases searched, search terms, and any filters or limits applied.
5. Study Selection Process:
Outline the process for screening and selecting studies, including how duplicates were handled and the number of reviewers involved.
6. Data Extraction:
Specify the data extraction process, including the data items collected from each selected study (e.g., author, publication year, study design, key findings).
7. Quality Assessment:
Explain how the quality or risk of bias of the included studies was assessed (e.g., using quality assessment tools or scales).
8. Data Synthesis:
Describe how the data from the selected studies were synthesized and analyzed. This may include narrative synthesis, meta-analysis, or thematic analysis.
9. Results:
Present the main findings of the systematic literature review, including key themes, trends, and conclusions drawn from the included studies.
10. Discussion: - Interpret the results in the context of the research question and objectives. Discuss the implications of the findings and any limitations of the review.
11. Conclusion: - Summarize the main contributions of the systematic literature review and provide recommendations for future research or practice.
12. References: - List all the studies included in the systematic literature review following a consistent citation style (e.g., APA, MLA).
13. Appendices: - Include any supplementary materials, such as flowcharts of the study selection process or data extraction forms.
14. Acknowledgments: - If applicable, acknowledge individuals or organizations that provided support or assistance during the review process.
This template can serve as a guide for conducting and documenting a systematic literature review in a structured and transparent manner. Adapt it to your specific research topic and requirements, and ensure that your systematic literature review adheres to established guidelines and best practices in the field.
Last updated 1 year ago
Run a free plagiarism check in 10 minutes, generate accurate citations for free.
Methodology
Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.
What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .
There are five key steps to writing a literature review:
A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.
Upload your document to correct all your mistakes in minutes
What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.
When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:
Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.
The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.
Try for free
Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.
You can also check out our templates with literature review examples and sample outlines at the links below.
Download Word doc Download Google doc
Before you begin searching for literature, you need a clearly defined topic .
If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .
Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.
Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:
You can also use boolean operators to help narrow down your search.
Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.
You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.
For each publication, ask yourself:
Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.
You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.
As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.
It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.
To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:
This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.
There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).
The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.
Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.
If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.
For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.
If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:
A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.
You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.
Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.
The introduction should clearly establish the focus and purpose of the literature review.
Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.
As you write, you can follow these tips:
In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.
When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !
This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.
Scribbr slides are free to use, customize, and distribute for educational purposes.
Open Google Slides Download PowerPoint
If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.
Statistics
Research bias
A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .
It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.
There are several reasons to conduct a literature review at the beginning of a research project:
Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.
The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .
A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .
An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .
If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.
McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved September 9, 2024, from https://www.scribbr.com/dissertation/literature-review/
Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".
I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”
JEPS Bulletin
The Official Blog of the Journal of European Psychology Students
Investigating concepts associated with psychology requires an indefinite amount of reading. Hence, good literature reviews are an inevitably needed part of providing the modern scientists with a broad spectrum of knowledge. In order to help, this blog post will introduce you to the basics of literature reviews and explain a specific methodological approach towards writing one, known as the systematic literature review.
Literature review is a term associated with the process of collecting, checking and (re)analysing data from the existing literature with a particular search question in mind. The latter could be for example:
A literature review (a) defines a specific issue, concept, theory, phenomena; (b) compiles published literature on a topic; (c) summarises critical points of current knowledge about the problem and (d) suggests next steps in addressing it.
Literature reviews can be based on all sorts of information found in scientific journals, books, academic dissertations, electronic bibliographic databases and the rest of the Internet. Electronic databases such as PsycINFO , PubMed , Web of Science could be a good starting point. Some of them, like EBSCOhost , ScienceDirect , SciELO , and ProQuest , provide full-text information, while others provide the users mostly with the abstracts of the material. Besides scientific literature, literature reviews often include the so called gray literature . This refers to the material that is either unpublished or published in non-commercial form (e.g., theses, dissertations, government reports, fact sheets, pre-prints of articles). Excluding it completely from a literature review is inappropriate because the search should be always as complete as possible in order to reduce the risk of publication bias. However, when reviewing the material on for example Google Scholar , Science.gov , Social Science Research Network , or PsycEXTRA it should be kept in mind that such search engines also display the material without peer-review and have therefore less credibility regarding the information they are disclosing.
When performing literature reviews, the use of appropriately selected terminology is essential, since it allows the researchers much clearer communication. In psychology, without some commonly agreed lists of terms, we would all get lost in the variety of concepts and vocabularies that could be applied. A typical recommendation for where to look for such index terms would be ‘ Thesaurus of Psychological Index Terms (2007) ’, which includes nearly 9,000 most commonly cross-referenced terms in psychology. In addition, electronic databases mentioned before sometimes prompt the use of the so-called Boolean operators , simple words such as AND, OR, NOT, or AND NOT. These are used for combining and/or excluding specific terms in your search and sometimes allow to obtain more focused and productive results in the search. Other tools to make search strategy more comprehensive and focused are also truncations – a tool for searching terminologies that have same initial roots (e.g., anxiety and anxious) and wildcards for words with spelling deviations (e.g., man and men). It is worth noting that the databases slightly differ in how they label the index terms and utilize specific search tools in their systems.
Among authors, there is not much coherence about different types of literature reviews but in general, most recognize at least two: traditional and systematic. The main difference between them is situated in the process of collecting and selecting data and the material for the review. Systematic literature review, as the name implies, is the more structured of the two and is thought to be more credible. On the other hand, traditional is thought to heavily depend on the researcher’s decisions regarding the data selection and, consequently, evaluation and results. Systematic protocol of the systematic literature review can be therefore understood as an optional solution for controlling the incomplete and possibly biased reports of traditional reviews.
The systematic literature review is a method/process/protocol in which a body of literature is aggregated, reviewed and assessed while utilizing pre-specified and standardized techniques. In other words, to reduce bias, the rationale, the hypothesis, and the methods of data collection are prepared before the review and are used as a guide for performing the process. Just like it is for the traditional literature reviews, the goal is to identify, critically appraise, and summarize the existing evidence concerning a clearly defined problem.
Systematic literature reviews allow us to examine conflicting and/or coincident findings, as well as to identify themes that require further investigation. Furthermore, they include the possibility of evaluating consistency and generalization of the evidence regarding specific scientific questions and are, therefore, also of great practical value within the psychological field. The method is particularly useful to integrate the information of a group of studies investigating the same phenomena and it typically focuses on a very specific empirical question, such as ‘Does the Rational Emotive Therapy intervention benefit the well-being of the patients diagnosed with depression?’.
Systematic literature reviews include all (or most) of the following characteristics:
The process of performing a systematic literature review consists of several stages and can be reported in a form of an original research article with the same name (i.e., systematic literature review):
1: Start by clearly defining the objective of the review or form a structured research question.
Place in the research article: Title, Abstract, Introduction.
Example of the objective: The objective of this literature revision is to systematically review and analyse the current research on the effects of music on the anxiety levels of children in hospital settings.
Example of a structured research question: What are the most important factors associated with the development of PTSD in soldiers?
Tip: In the title, identify that the report is a systematic literature review.
2: Clearly specify the methodology of the review and define eligibility criteria (i.e., study selection criteria that the published material must meet in order to be included or excluded from the study). The search should be extensive.
Place in the research article: Methods.
Examples of inclusion criteria: Publication was an academic and peer-reviewed study. Publication was a study that examined the effects of regular physical exercise intervention on depression and included a control group.
Examples of exclusion criteria: Publication was involving male adults. Studies that also examined non-physical activities as interventions. Studies that were only published in a language other than English.
Tips: The eligibility criteria sometimes fit to be presented in tables.
3: Retrieve eligible literature and thoroughly report your search strategy throughout the process. (Ideally, the selection process is performed by at least two independent investigators.)
Example: The EBSCOhost and PsychInfo electronic databases from 2010 to 2017 were searched. These were chosen because of the psychological focus that encompasses psychosocial effects of emotional abuse in childhood. Search terms were ‘emotional abuse’, ‘childhood’, ‘psychosocial effects’, and ‘psychosocial consequences’. The EBSCOhost produced 200 results from the search criteria, while PsychInfo produced 467, for a total of 667 articles. […] Articles were rejected if it was determined from the title and the abstract that the study failed to meet the inclusion criteria. Any ambiguities regarding the application of the selection criteria were resolved through discussions between all the researchers involved.
Tip: Sometimes it is nice to represent the selection process in a graphical representation; in the form of a decision tree or a flow diagram (check PRISMA ).
4: Assess the methodological quality of the selected literature whenever possible and exclude the articles with low methodological quality. Keep in mind that the quality of the systematic review depends on the validity and the quality of the studies included in the review.
Examples of the instruments available for evaluating the quality of the studies: PEDro, Jadad scale, the lists of Delphi, OTseeker, Maastricht criteria.
Tip: Present the excluded articles as a part of the selection process mentioned in step 3.
5: Proceed with the so-called characterization of the studies. Decide which data to look for in all the selected studies and present it in a summarized way. If the information is missing in some specific paper, always register it in your reports. (Ideally, the characterization of the studies is performed by at least two independent investigators.)
Place in the research article: Results.
Examples of the information that should and/or could be collected for characterization of the literature: authors, year, sample size, study design, aims and objectives, findings/results, limitations.
Tip: Sometimes results can be presented nicely in a form of a table depicting the main characteristics.
6: Write a synthesis of the results – integrate the results of different studies and interpret them in a narrative form.
Place in the research article: Interpretation, Conclusions.
Patterns discovered as results should be summarized in a qualitative, narrative form. Modulate one (or more) general arguments for organizing the review. Some trick to help you do this is to choose two or three main information sources (e.g., articles, books, other literature reviews) to explain the results of other studies through a similar way of organization. Connect the information reported by different sources and do not just summarize the results. Find patterns in the results of different studies, identify them, address the theoretical and/or methodological conflicts and try to interpret them. Summarize the principal conclusions and evaluate the current state on the subject by pointing out possible further directions.
The results emerging from the data that were included in such retrospective studies can lead to a certain level of credibility regarding their conclusions. Actually, systematic literature reviews are thought to be one of our best methods to summarize and synthesize evidence about some specific research question and are often used as the main ‘practice making guidelines’ in many health care disciplines. Therefore, it is no wonder why systematic reviews are gaining popularity among researchers and why journals are moving in this direction as well. This also shows in the development of more and more specific guidelines and checklists for writing systematic literature reviews (see for example PRISMA or Cochrane Handbook for Systematic Reviews of Interventions ). To find examples of systematic literature review articles you can check Cochrane Database of Systematic Reviews , BioMed Central’s Systematic Reviews Journal , and PROSPERO . If you are aware of the concept of ‘registered reports’, it is worth mentioning that submitting with PROSPERO provides you with the option of publishing the latter as well. I suggest that you go through the list of useful resources provided below and hopefully, you can get enough information about anything related that remained unanswered. Now, I encourage you to try to be a little more to be systematic whenever researching some topic, to try to write a systematic literature review yourself and to maybe even consider submitting it to JEPS .
EBSCOhost : https://search.ebscohost.com/
Google Scholar : https://scholar.google.com/
PRISMA : http://www.prisma-statement.org/
PROSPERO : https://www.crd.york.ac.uk/prospero/
ProQuest : http://www.proquest.com/
PsycEXTRA : http://www.apa.org/pubs/databases/psycextra/index.aspx :
PsycINFO : http://www.apa.org/pubs/databases/psycinfo/index.aspx
PubMed : https://www.ncbi.nlm.nih.gov/pubmed/
SciELO : http://www.scielo.org/php/index.php?lang=en
Science.gov : https://www.science.gov/
ScienceDirect : http://www.sciencedirect.com/
Scorpus : http://www.scopus.com/freelookup/form/author.uri
Social Science Research Network : https://www.ssrn.com/en/
Systematic Reviews Journal (BIOMED) : https://systematicreviewsjournal.biomedcentral.com/
Web of Science : https://webofknowledge.com/
Other sources
Eva Štrukelj is currently studying Clinical and Health Psychology at the University of Algarve in Portugal. Her main areas of interest are social psychology and health psychology. Regarding research, she is particularly curious about stigma and with it related topics.
Related posts:.
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Email citation, add to collections.
Your saved search, create a file for external citation management software, your rss feed.
Affiliations.
This article provides a step-by-step approach to conducting and reporting systematic literature reviews (SLRs) in the domain of healthcare design and discusses some of the key quality issues associated with SLRs. SLR, as the name implies, is a systematic way of collecting, critically evaluating, integrating, and presenting findings from across multiple research studies on a research question or topic of interest. SLR provides a way to assess the quality level and magnitude of existing evidence on a question or topic of interest. It offers a broader and more accurate level of understanding than a traditional literature review. A systematic review adheres to standardized methodologies/guidelines in systematic searching, filtering, reviewing, critiquing, interpreting, synthesizing, and reporting of findings from multiple publications on a topic/domain of interest. The Cochrane Collaboration is the most well-known and widely respected global organization producing SLRs within the healthcare field and a standard to follow for any researcher seeking to write a transparent and methodologically sound SLR. Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), like the Cochrane Collaboration, was created by an international network of health-based collaborators and provides the framework for SLR to ensure methodological rigor and quality. The PRISMA statement is an evidence-based guide consisting of a checklist and flowchart intended to be used as tools for authors seeking to write SLR and meta-analyses.
Keywords: evidence based design; healthcare design; systematic literature review.
PubMed Disclaimer
Full text sources.
NCBI Literature Resources
MeSH PMC Bookshelf Disclaimer
The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .
Angela carrera-rivera.
a Faculty of Engineering, Mondragon University
Felix larrinaga.
b Design Innovation Center(DBZ), Mondragon University
Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:
Specifications table
Subject area: | Computer-science |
More specific subject area: | Software engineering |
Name of your method: | Systematic literature review |
Name and reference of original method: | |
Resource availability: | Resources referred to in this article: ) ) |
A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.
Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.
The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.
Planning Step 1 “Defining PICOC keywords and synonyms”.
Description | Example (PICOC) | Example (Synonyms) | |
---|---|---|---|
Population | Can be a specific role, an application area, or an industry domain. | Smart Manufacturing | • Digital Factory • Digital Manufacturing • Smart Factory |
Intervention | The methodology, tool, or technology that addresses a specific issue. | Semantic Web | • Ontology • Semantic Reasoning |
Comparison | The methodology, tool, or technology in which the is being compared (if appropriate). | Machine Learning | • Supervised Learning • Unsupervised Learning |
Outcome | Factors of importance to practitioners and/or the results that could produce. | Context-Awareness | • Context-Aware • Context-Reasoning |
Context | The context in which the comparison takes place. Some systematic reviews might choose to exclude this element. | Business Process Management | • BPM • Business Process Modeling |
Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).
Research questions examples.
Research Questions examples |
---|
• : What are the current challenges of context-aware systems that support the decision-making of business processes in smart manufacturing? • : Which technique is most appropriate to support decision-making for business process management in smart factories? • : In which scenarios are semantic web and machine learning used to provide context-awareness in business process management for smart manufacturing? |
The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.
Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.
Database | Description | URL | Area | Advanced Search Y/N |
---|---|---|---|---|
Scopus | From Elsevier. sOne of the largest databases. Very user-friendly interface | Interdisciplinary | Y | |
Web of Science | From Clarivate. Multidisciplinary database with wide ranging content. | Interdisciplinary | Y | |
EI Compendex | From Elsevier. Focused on engineering literature. | Engineering | Y (Query view not available) | |
IEEE Digital Library | Contains scientific and technical articles published by IEEE and its publishing partners. | Engineering and Technology | Y | |
ACM Digital Library | Complete collection of ACM publications. | Computing and information technology | Y |
Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.
Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.
Criteria Type | Description | Example |
---|---|---|
Period | Articles can be selected based on the time period to review, e.g., reviewing the technology under study from the year it emerged, or reviewing progress in the field since the publication of a prior literature review. | : From 2015 to 2021 Articles prior 2015 |
Language | Articles can be excluded based on language. | : Articles not in English |
Type of Literature | Articles can be excluded if they are fall into the category of grey literature. | Reports, policy literature, working papers, newsletters, government documents, speeches |
Type of source | Articles can be included or excluded by the type of origin, i.e., conference or journal articles or books. | : Articles from Conferences or Journals Articles from books |
Impact Source | Articles can be excluded if the author limits the impact factor or quartile of the source. | Articles from Q1, and Q2 sources : Articles with a Journal Impact Score (JIS) lower than |
Accessibility | Not accessible in specific databases. | : Not accessible |
Relevance to research questions | Articles can be excluded if they are not relevant to a particular question or to “ ” number of research questions. | Not relevant to at least 2 research questions |
Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.
Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.
Do the researchers discuss any problems (limitations, threats) with the validity of their results (reliability)? | 1 – No, and not considered (Score: 0) 2 – Partially (Score: 0.5) 3 – Yes (Score: 1) |
Is there a clear definition/ description/ statement of the aims/ goals/ purposes/ motivations/ objectives/ questions of the research? | 1 – Disagree (Score: 1) 2 – Somewhat disagree (Score: 2) 3 – Neither agree nor disagree (Score: 3) 4 – Somewhat agree (Score: 4) 5 – Agree (Score: 5) |
The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .
Planning Step 6 “Define data extraction form”. Examples of fields.
Classification and fields to consider for data extraction | Description and examples |
---|---|
Research type | • focuses on abstract ideas, concepts, and theories built on literature reviews . • uses scientific data or case studies for explorative, descriptive, explanatory, or measurable findings . an SLR on context-awareness for S-PSS and categorized the articles in theoretical and empirical research. |
By process phases, stages | When analyzing a process or series of processes, an effective way to structure the data is to find a well-established framework of reference or architecture. : • an SLR on self-adaptive systems uses the MAPE-K model to understand how the authors tackle each module stage. • presented a context-awareness survey using the stages of context-aware lifecycle to review different methods. |
By technology, framework, or platform | When analyzing a computer science topic, it is important to know the technology currently employed to understand trends, benefits, or limitations. : • an SLR on the big data ecosystem in the manufacturing field that includes frameworks, tools, and platforms for each stage of the big data ecosystem. |
By application field and/or industry domain | If the review is not limited to a specific “Context” or “Population" (industry domain), it can be useful to identify the field of application : • an SLR on adaptive training using virtual reality (VR). The review presents an extensive description of multiple application domains and examines related work. |
Gaps and challenges | Identifying gaps and challenges is important in reviews to determine the research needs and further establish research directions that can help scholars act on the topic. |
Findings in research | Research in computer science can deliver multiple types of findings, e.g.: |
Evaluation method | Case studies, experiments, surveys, mathematical demonstrations, and performance indicators. |
The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.
After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.
Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:
(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”
Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.
Example of Advanced search on Scopus.
In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.
The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.
In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.
Performing quality assessment (QA) in Parsif.al.
Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.
Example of data extraction form using Parsif.al.
The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.
The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.
The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.
Techniques for bibliometric analysis and examples.
Publication-related analysis | Description | Example |
---|---|---|
Years of publications | Determine interest in the research topic by years or the period established by the SLR, by quantifying the number of papers published. Using this information, it is also possible to forecast the growth rate of research interest. | [ ] identified the growth rate of research interest and the yearly publication trend. |
Top contribution journals/conferences | Identify the leading journals and conferences in which authors can share their current and future work. | , |
Top countries' or affiliation contributions | Examine the impacts of countries or affiliations leading the research topic. | [ , ] identified the most influential countries. |
Leading authors | Identify the most significant authors in a research field. | - |
Keyword correlation analysis | Explore existing relationships between topics in a research field based on the written content of the publication or related keywords established in the articles. | using keyword clustering analysis ( ). using frequency analysis. |
Total and average citation | Identify the most relevant publications in a research field. | Scatter plot citation scores and journal factor impact |
Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.
[1] Keyword co-relationship analysis using clusterization in vos viewer.
This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.
Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.
The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].
SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.
Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.
Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.
1 https://parsif.al/
A systematic literature review (SLR) is an independent academic method that aims to identify and evaluate all relevant literature on a topic in order to derive conclusions about the question under consideration. "Systematic reviews are undertaken to clarify the state of existing research and the implications that should be drawn from this." (Feak & Swales, 2009, p. 3) An SLR can demonstrate the current state of research on a topic, while identifying gaps and areas requiring further research with regard to a given research question. A formal methodological approach is pursued in order to reduce distortions caused by an overly restrictive selection of the available literature and to increase the reliability of the literature selected (Tranfield, Denyer & Smart, 2003). A special aspect in this regard is the fact that a research objective is defined for the search itself and the criteria for determining what is to be included and excluded are defined prior to conducting the search. The search is mainly performed in electronic literature databases (such as Business Source Complete or Web of Science), but also includes manual searches (reviews of reference lists in relevant sources) and the identification of literature not yet published in order to obtain a comprehensive overview of a research topic.
An SLR protocol documents all the information gathered and the steps taken as part of an SLR in order to make the selection process transparent and reproducible. The PRISMA flow-diagram support you in making the selection process visible.
In an ideal scenario, experts from the respective research discipline, as well as experts working in the relevant field and in libraries, should be involved in setting the search terms . As a rule, the literature is selected by two or more reviewers working independently of one another. Both measures serve the purpose of increasing the objectivity of the literature selection. An SLR must, then, be more than merely a summary of a topic (Briner & Denyer, 2012). As such, it also distinguishes itself from “ordinary” surveys of the available literature. The following table shows the differences between an SLR and an “ordinary” literature review.
Characteristic | SLR | common literature overview |
---|---|---|
Independent research method | yes | no |
Explicit formulation of the search objectives | yes | no |
Identification of all publications on a topic | yes | no |
Defined criteria for inclusion and exclusion of publications | yes | no |
Description of search procedure | yes | no |
Literature selection and information extraction by several persons | yes | no |
Transparent quality evaluation of publications | yes | no |
A SLR has several process steps which are defined differently in the literature (Fink 2014, p. 4; Guba 2008, Transfield et al. 2003). We distinguish the following steps which are adapted to the economics and management research area:
Briner & Denyer (2009, p. 347ff.) have developed the CIMO scheme to establish clearly formulated and answerable research questions in the field of economic sciences:
C – CONTEXT: Which individuals, relationships, institutional frameworks and systems are being investigated?
I – Intervention: The effects of which event, action or activity are being investigated?
M – Mechanisms: Which mechanisms can explain the relationship between interventions and results? Under what conditions do these mechanisms take effect?
O – Outcomes: What are the effects of the intervention? How are the results measured? What are intended and unintended effects?
The objective of the systematic literature review is used to formulate research questions such as “How can a project team be led effectively?”. Since there are numerous interpretations and constructs for “effective”, “leadership” and “project team”, these terms must be particularized.
With the aid of the scheme, the following concrete research questions can be derived with regard to this example:
Under what conditions (C) does leadership style (I) influence the performance of project teams (O)?
Which constructs have an effect upon the influence of leadership style (I) on a project team’s performance (O)?
Research questions do not necessarily need to follow the CIMO scheme, but they should:
As early as this stage, the criteria for inclusion and exclusion are also defined. The selection of the criteria must be well-grounded. This may include conceptual factors such as a geographical or temporal restrictions, congruent definitions of constructs, as well as quality criteria (journal impact factor > x).
The selection of sources must be described and explained in detail. The aim is to find a balance between the relevance of the sources (content-related fit) and the scope of the sources.
In the field of economic sciences, there are a number of literature databases that can be searched as part of an SLR. Some examples in this regard are:
Our video " Selecting the right databases " explains how to find relevant databases for your topic.
Literature databases are an important source of research for SLRs, as they can minimize distortions caused by an individual literature selection (selection bias), while offering advantages for a systematic search due to their data structure. The aim is to find all database entries on a topic and thus keep the retrieval bias low (tutorial on retrieval bias ). Besides articles from scientific journals, it is important to inlcude working papers, conference proceedings, etc to reduce the publication bias ( tutorial on publication bias ).
Our online self-study course " Searching economic databases " explains step 2 und 3.
Once the literature databases and other research sources have been selected, search terms are defined. For this purpose, the research topic/questions is/are divided into blocks of terms of equal ranking. This approach is called the block-building method (Guba 2008, p. 63). The so-called document-term matrix, which lists topic blocks and search terms according to a scheme, is helpful in this regard. The aim is to identify as many different synonyms as possible for the partial terms. A precisely formulated research question facilitates the identification of relevant search terms. In addition, keywords from particularly relevant articles support the formulation of search terms.
A document-term matrix for the topic “The influence of management style on the performance of project teams” is shown in this example .
Identification of headwords and keywords
When setting search terms, a distinction must be made between subject headings and keywords, both of which are described below:
Subject headings
Subject headings are a standardized list of words that are generated by the specialists in charge of some databases. This so-called index of subject headings (thesaurus) helps searchers find relevant articles, since the headwords indicate the content of a publication. By contrast, an ordinary keyword search does not necessarily result in a content-related fit, since the database also displays articles in which, for example, a word appears once in the abstract, even though the article’s content does not cover the topic.
Nevertheless, searches using both headwords and keywords should be conducted, since some articles may not yet have been assigned headwords, or errors may have occurred during the assignment of headwords.
To add headwords to your search in the Business Source Complete database, please select the Thesaurus tab at the top. Here you can find headwords in a new search field and integrate them into your search query. In the search history, headwords are marked with the addition DE (descriptor).
The EconBiz database of the German National Library of Economics (ZBW – Leibniz Information Centre for Economics), which also contains German-language literature, has created its own index of subject headings with the STW Thesaurus for Economics . Headwords are integrated into the search by being used in the search query.
Since the indexes of subject headings divide terms into synonyms, generic terms and sub-aspects, they facilitate the creation of a document-term matrix. For this purpose it is advisable to specify in the document-term matrix the origin of the search terms (STW Thesaurus for Economics, Business Source Complete, etc.).
Searching in literature databases
Once the document-term matrix has been defined, the search in literature databases begins. It is recommended to enter each word of the document-term matrix individually into the database in order to obtain a good overview of the number of hits per word. Finally, all the words contained in a block of terms are linked with the Boolean operator OR and thereby a union of all the words is formed. The latter are then linked with each other using the Boolean operator AND. In doing so, each block should be added individually in order to see to what degree the number of hits decreases.
Since the search query must be set up separately for each database, tools such as LitSonar have been developed to enable a systematic search across different databases. LitSonar was created by Professor Dr. Ali Sunyaev (Institute of Applied Informatics and Formal Description Methods – AIFB) at the Karlsruhe Institute of Technology.
Advanced search
Certain database-specific commands can be used to refine a search, for example, by taking variable word endings into account (*) or specifying the distance between two words, etc. Our overview shows the most important search commands for our top databases.
Additional searches in sources other than literature databases
In addition to literature databases, other sources should also be searched. Fink (2014, p. 27) lists the following reasons for this:
Therefore, further search strategies are manual search, bibliographic analysis, personal contacts and academic networks (Briner & Denyer, p. 349). Manual search means that you go through the source information of relevant articles and supplement your hit list accordingly. In addition, you should conduct a targeted search for so-called gray literature, that is, literature not distributed via the book trade, such as working papers from specialist areas and conference reports. By including different types of publications, the so-called publication bias (DBWM video “Understanding publication bias” ) – that is, distortions due to exclusive use of articles from peer-reviewed journals – should be kept to a minimum.
The PRESS-Checklist can support you to check the correctness of your search terms.
In principle, large amounts of data can be easily collected, structured and sorted with data processing programs such as Excel. Another option is to use reference management programs such as EndNote, Citavi or Zotero. The Saxon State and University Library Dresden (SLUB Dresden) provides an overview of current reference management programs . Software for qualitative data analysis such as NVivo is equally suited for data processing. A comprehensive overview of the features of different tools that support the SLR process can be found in Bandara et al. (2015).
Our online-self study course "Managing literature with Citavi" shows you how to use the reference management software Citavi.
When conducting an SLR, you should specify for each hit the database from which it originates and the date on which the query was made. In addition, you should always indicate how many hits you have identified in the various databases or, for example, by manual search.
Exporting data from literature databases
Exporting from literature databases is very easy. In Business Source Complete , you must first click on the “Share” button in the hit list, then “Email a link to download exported results” at the very bottom and then select the appropriate format for the respective literature program.
Exporting data from the literature database EconBiz is somewhat more complex. Here you must first create a marked list and then select each hit individually and add it to the marked list. Afterwards, articles on the list can be exported.
After merging all hits from the various databases, duplicate entries (duplicates) are deleted.
All publications are evaluated in the literature management program applying the previously defined criteria for inclusion and exclusion. Only those sources that survive this selection process will subsequently be analyzed. The review process and inclusion criteria should be tested with a small sample and adjustments made if necessary before applying it to all articles. In the ideal case, even this selection would be carried out by more than one person, with each working independently of one another. It needs to be made clear how discrepancies between reviewers are dealt with.
The review of the criteria for inclusion and exclusion is primarily based on the title, abstract and subject headings in the databases, as well as on the keywords provided by the authors of a publication in the first step. In a second step the whole article / source will be read.
You can create tag words for the inclusion and exclusion in your literature management tool to keep an overview.
In addition to the common literature management tools, you can also use software tools that have been developed to support SLRs. The central library of the university in Zurich has published an overview and evaluation of different tools based on a survey among researchers. --> View SLR tools
The selection process needs to be made transparent. The PRISMA flow diagram supports the visualization of the number of included / excluded studies.
Forward and backward search
Should it become apparent that the number of sources found is relatively small, or if you wish to proceed with particular thoroughness, a forward-and-backward search based on the sources found is recommendable (Webster & Watson 2002, p. xvi). A backward search means going through the bibliographies of the sources found. A forward search, by contrast, identifies articles that have cited the relevant publications. The Web of Science and Scopus databases can be used to perform citation analyses.
As the next step, the remaining titles are analyzed as to their content by reading them several times in full. Information is extracted according to defined criteria and the quality of the publications is evaluated. If the data extraction is carried out by more than one person, a training ensures that there will be no differences between the reviewers.
Depending on the research questions there exist diffent methods for data abstraction (content analysis, concept matrix etc.). A so-called concept matrix can be used to structure the content of information (Webster & Watson 2002, p. xvii). The image to the right gives an example of a concept matrix according to Becker (2014).
Particularly in the field of economic sciences, the evaluation of a study’s quality cannot be performed according to a generally valid scheme, such as those existing in the field of medicine, for instance. Quality assessment therefore depends largely on the research questions.
Based on the findings of individual studies, a meta-level is then applied to try to understand what similarities and differences exist between the publications, what research gaps exist, etc. This may also result in the development of a theoretical model or reference framework.
Article | Pattern | Configuration | Similarities |
---|---|---|---|
Thom (2008) | x | ||
Yang (2009) | x | x | |
Rosa (2009) | x | x |
Once the review has been conducted, the results must be compiled and, on the basis of these, conclusions derived with regard to the research question (Fink 2014, p. 199ff.). This includes, for example, the following aspects:
Bandara, W., Furtmueller, E., Miskon, S., Gorbacheva, E., & Beekhuyzen, J. (2015). Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support. Communications of the Association for Information Systems . 34(8), 154-204.
Booth, A., Papaioannou, D., and Sutton, A. (2012) Systematic approaches to a successful literature review. London: Sage.
Briner, R. B., & Denyer, D. (2012). Systematic Review and Evidence Synthesis as a Practice and Scholarship Tool. In Rousseau, D. M. (Hrsg.), The Oxford Handbook of Evidenence Based Management . (S. 112-129). Oxford: Oxford University Press.
Durach, C. F., Wieland, A., & Machuca, Jose A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review . International Journal of Physical Distribution & Logistic Management , 46 (1/2), 118-137. doi: https://doi.org/10.1108/IJPDLM-05-2013-0133
Feak, C. B., & Swales, J. M. (2009). Telling a Research Story: Writing a Literature Review. English in Today's Research World 2. Ann Arbor: University of Michigan Press. doi: 10.3998/mpub.309338
Fink, A. (2014). Conducting Research Literature Reviews: From the Internet to Paper (4. Aufl.). Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.
Fisch, C., & Block, J. (2018). Six tips for your (systematic) literature review in business and management research. Management Review Quarterly, 68, 103–106 (2018). doi.org/10.1007/s11301-018-0142-x
Guba, B. (2008). Systematische Literaturrecherche. Wiener Medizinische Wochenschrift , 158 (1-2), S. 62-69. doi: doi.org/10.1007/s10354-007-0500-0 Hart, C. Doing a literature review: releasing the social science research imagination. London: Sage.
Jesson, J. K., Metheson, L. & Lacey, F. (2011). Doing your Literature Review - traditional and Systematic Techniques . Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71.
Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide . Oxford:Blackwell. Ridley, D. (2012). The literature review: A step-by-step guide . 2nd edn. London: Sage.
Chang, W. and Taylor, S.A. (2016), The Effectiveness of Customer Participation in New Product Development: A Meta-Analysis, Journal of Marketing , American Marketing Association, Los Angeles, CA, Vol. 80 No. 1, pp. 47–64.
Tranfield, D., Denyer, D. & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management , 14 (3), S. 207-222. doi: https://doi.org/10.1111/1467-8551.00375
Webster, J., & Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review. Management Information Systems Quarterly , 26(2), xiii-xxiii. http://www.jstor.org/stable/4132319
Durach, C. F., Wieland, A. & Machuca, Jose. A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review. International Journal of Physical Distribution & Logistics Management, 45(1/2), 118 – 137.
What is particularly good about this example is that search terms were defined by a number of experts and the review was conducted by three researchers working independently of one another. Furthermore, the search terms used have been very well extracted and the procedure of the literature selection very well described.
On the downside, the restriction to English-language literature brings the language bias into play, even though the authors consider it to be insignificant for the subject area.
Bos-Nehles, A., Renkema, M. & Janssen, M. (2017). HRM and innovative work behaviour: a systematic literature review. Personnel Review, 46(7), pp. 1228-1253
Jia, F., Orzes, G., Sartor, M. & Nassimbeni, G. (2017). Global sourcing strategy and structure: towards a conceptual framework. International Journal of Operations & Production Management, 37(7), 840-864
Franziska Klatt
+49 30 314-29778
Privacy notice: The TU Berlin offers a chat information service. If you enable it, your IP address and chat messages will be transmitted to external EU servers. more information
The chat is currently unavailable.
Please use our alternative contact options.
The librarian plays an integral role in systematic reviews at Loma Linda University.
Cochrane Reviews provides the following definition for a systematic review: "A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting systematic reviews use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making."
A systematic review is a rigorous and comprehensive approach to reviewing and synthesizing existing research literature on a specific topic. It goes beyond a traditional literature review by using a systematic and transparent process to identify, select, appraise, and analyze relevant studies.
The purpose of a systematic review is to provide a reliable and unbiased summary of the available evidence on a particular research question or topic. By systematically searching for and critically evaluating all relevant studies, systematic reviews aim to minimize bias and provide a more objective assessment of the existing evidence.
Systematic reviews are essential in research for several reasons:
Evidence-based decision making
Summarizing complex bodies of evidence
Identifying research gaps and priorities
Resolving conflicting findings
Improving research efficiency
To request a systematic review service, contact the jbi certified librarians below: .
liaison to the school of allied health professions, and the school of public health.
office (909) 558-1000 ext. 47564 · e-mail [email protected]
Make an appointment with Adorée
and the school of nursing (undergraduate).
office: (909) 558-1000 ext. 47561 e-mail: [email protected]
library director.
office: (909) 558-1000 ext. 47501
e-mail: [email protected]
Discover the world's research
Search the Library for Articles, Books, and More Find It!
Pro tip: Save time and stress by piloting your data extraction form with the whole team. If there are any fields missing or unclear, it's much easier to fix them now.
In your protocol.
Your protocol includes a description of how you will extract data from your screened and appraised articles.
Some elements you will need to decide about data extraction plan include:
For a full description of the data extraction step, see Chapter 5 in the Cochrane Handbook for Systematic Reviews of Interventions or one of the other resources below:
You should plan to extract only data that is relevant to answering the question posed in your systematic review. It may help to consult other similar systematic reviews to identify what data to collect. You should use your key questions and your eligibility criteria as a starting point. It can also help to think about your question in a framework such as PICO .
If you plan to synthesize data, you will want to collect additional information such as sample sizes, effect sizes, dependent variables, reliability measures, pre-test data, post-test data, follow-up data, and statistical tests used.
Data extraction tips.
Login to LibApps
Email us! [email protected]
Text Us! (616) 818-0219
Mary Idema Pew Library (616) 331-3500
Steelcase Library (616) 331-6799
Frey Foundation Learning Commons (616) 331-5933
IMAGES
VIDEO
COMMENTS
Learn how to conduct and write a systematic literature review in nine steps with this guide from Paperpile. Find out the types, purposes, and best practices of systematic reviews.
This article aims to provide an overview of the structure, form and content of systematic reviews. It focuses in particular on the literature searching component, and covers systematic database searching techniques, searching for grey literature and the ...
A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.
Abstract This article aims to provide an overview of the structure, form and content of systematic reviews. It focuses in particular on the literature searching component, and covers systematic database searching techniques, searching for grey literature and the importance of librarian involvement in the search. It also covers systematic review reporting standards such as PRISMA-P and PRISMA ...
This article provides a comprehensive guide on how to write a systematic review, a type of literature review that summarizes and synthesizes existing evidence on a topic.
Guidelines for writing a systematic review. 1. Introduction. A key feature of any academic activity is to have a sufficient understanding of the subject area under investigation and thus an awareness of previous research. Undertaking a literature review with an analysis of the results on a specific issue is required to demonstrate sufficient ...
Abstract Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer ...
Systematic review manuscript template (.doc) adapted from the PRISMA 2020 checklist. This document provides authors with template for writing about their systematic review.
The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information.
Abstract Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in ...
Resources for conducting a systematic review research.
A Systematic Literature Review (SLR) template is a structured framework used for conducting and documenting a systematic review of existing research studies on a specific topic or research question.
A literature review is a survey of scholarly knowledge on a topic. Our guide with examples, video, and templates can help you write yours.
The systematic literature review is a method/process/protocol in which a body of literature is aggregated, reviewed and assessed while utilizing pre-specified and standardized techniques. In other words, to reduce bias, the rationale, the hypothesis, and the methods of data collection are prepared before the review and are used as a guide for ...
This article provides a step-by-step approach to conducting and reporting systematic literature reviews (SLRs) in the domain of healthcare design and discusses some of the key quality issues associated with SLRs. SLR, as the name implies, is a systematic way of collecting, critically evaluating, integrating, and presenting findings from across ...
How to Write a Systematic Review: A Step-by-Step Guide. 1Sarah M. Yannascoli, MD 1Mara L. Schenker, MD 1James L. Carey, MD, MPH 1Jaimo Ahn, MD, PhD 1,2Keith D. Baldwin MD, MSPT, MPH. Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, PA. spital of Philadelphia, Philadelphia, PAIntroductionA systematic review attempts ...
Systematic Review Protocols Creating a systematic review protocol is an important step in the planning process for your review. A review protocol is beneficial for a number of reasons: It helps to ensure that all team members are on the same page when it comes to the research question, inclusion/exclusion criteria, etc.
ABSTRACT Systematic literature reviews (SRs) are a way of synthesising scientific evidence to answer a particular research question in a way that is transparent and reproducible, while seeking to include all published evidence on the topic and appraising the quality of this evidence. SRs have become a major methodology in disciplines such as public policy research and health sciences. Some ...
Abstract. Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in ...
Writing a Systematic Literature Review: Resources for Students and TraineesThis resource provides basic guidance and links to r. sources that will help when planning a systematic review of the literature. It does not replace guidance from y. ur research project supervisors and your university or hospital librarians. systematic lite. uestion ...
Welcome to the NEW Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) website Here you can access information about the PRISMA reporting guidelines, which are designed to help authors transparently report why their systematic review was done, what methods they used, and what they found.
What is a Systematic Literature Review? A systematic literature review (SLR) is an independent academic method that aims to identify and evaluate all relevant literature on a topic in order to derive conclusions about the question under consideration. "Systematic reviews are undertaken to clarify the state of existing research and the ...
It goes beyond a traditional literature review by using a systematic and transparent process to identify, select, appraise, and analyze relevant studies. The purpose of a systematic review is to provide a reliable and unbiased summary of the available evidence on a particular research question or topic. By systematically searching for and ...
This report to evaluated like example five articles marked like systematic literature review in relation with high performance computational engineering systems, being hardware or software based.
Grey Literature ; Record and Report your Search Strategy ; 5. Organize & Screen Search Results ... You should plan to extract only data that is relevant to answering the question posed in your systematic review. It may help to consult other similar systematic reviews to identify what data to collect. ... you will want to collect additional ...
Insight refers to a person's understanding of themselves and the world around them. Recent literature has explored people's insight into their substance use disorder (SUD) and how this is linked to treatment adherence, abstinence rates, and comorbid mental health symptoms. The aim of this systematic review was to synthesise and critically examine the existing literature on insight in SUD.
The literature was systematically reviewed following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to determine the effect of the HLA-DR profile on OM susceptibility. ... the NIH risk of bias tool revealed that most studies included did not justify their sample size, or the assessors were not blinded. Of all the ...