literature review of surveys

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

Search for relevant literature
Evaluate sources
Identify themes, debates, and gaps
Outline the structure
Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

Quick Run-through
Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

Demonstrate your familiarity with the topic and its scholarly context
Develop a theoretical framework and methodology for your research
Position your work in relation to other researchers and theorists
Show how your research addresses a gap or contributes to a debate
Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

Academic style
Vague sentences
Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
Body image, self-perception, self-esteem, mental health
Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

Your university’s library catalogue
Google Scholar
Project Muse (humanities and social sciences)
Medline (life sciences and biomedicine)
EconLit (economics)
Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

What question or problem is the author addressing?
What are the key concepts and how are they defined?
What are the key theories, models, and methods?
Does the research use established frameworks or take an innovative approach?
What are the results and conclusions of the study?
How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
Themes: what questions or concepts recur across the literature?
Debates, conflicts and contradictions: where do sources disagree?
Pivotal publications: are there any influential theories or studies that changed the direction of the field?
Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

Most research has focused on young women.
There is an increasing interest in the visual aspects of social media.
But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

Look at what results have emerged in qualitative versus quantitative research
Discuss how the topic has been approached by empirical versus theoretical scholarship
Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
Critically evaluate: mention the strengths and weaknesses of your sources
Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

Sampling methods
Simple random sampling
Stratified sampling
Cluster sampling
Likert scales
Reproducibility

Statistics

Null hypothesis
Statistical power
Probability distribution
Effect size
Poisson distribution

Research bias

Optimism bias
Cognitive bias
Implicit bias
Hawthorne effect
Anchoring bias
Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

To familiarize yourself with the current state of knowledge on your topic
To ensure that you’re not just repeating what others have already done
To identify gaps in knowledge and unresolved problems that your research can address
To develop your theoretical framework and methodology
To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .

An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved September 3, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process.

What is the purpose of literature review , a. habitat loss and species extinction: , b. range shifts and phenological changes: , c. ocean acidification and coral reefs: , d. adaptive strategies and conservation efforts: .

Choose a Topic and Define the Research Question:
Decide on the Scope of Your Review:
Select Databases for Searches:
Conduct Searches and Keep Track:
Review the Literature:
Organize and Write Your Literature Review:
How to write a literature review faster with Paperpal?

Frequently asked questions

What is a literature review .

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2

1. Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge.

2. Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field.

Find academic papers related to your research topic faster. Try Research on Paperpal

3. Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research.

4. Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered.

5. Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research.

6. Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature.

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic.

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements.

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources.

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems.

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning.

Strengthen your literature review with factual insights. Try Research on Paperpal for free!

How to write a good literature review

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements.

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review.

Opening Statement: Begin with a general statement about the broader topic and its significance in the field.
Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address.
Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research.
Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic.
Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review.

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria.

Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement.
Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic.
Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature.
Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps.
Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review.

Write and Cite as yo u go with Paperpal Research. Start now for free!

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research.

Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective.
Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field.
Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice.
Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field.
Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community.

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1

Choose a Topic and Define the Research Question:

Select a topic that is relevant to your field of study.
Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore?

Decide on the Scope of Your Review:

Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview?
Consider the geographical scope. Is your review global, or are you focusing on a specific region?
Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude?

Select Databases for Searches:

Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar.
Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic.

Conduct Searches and Keep Track:

Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques.
Record and document your search strategy for transparency and replicability.
Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references.

Review the Literature:

Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies.
Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research.
Summarize key findings and arguments from each source. Compare and contrast different perspectives.
Identify areas where there is a consensus in the literature and where there are conflicting opinions.
Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research?

Organize and Write Your Literature Review:

Literature review outline should be based on themes, chronological order, or methodological approaches.
Write a clear and coherent narrative that synthesizes the information gathered.
Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.).
Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research.

Whether you’re exploring a new research field or finding new angles to develop an existing topic, sifting through hundreds of papers can take more time than you have to spare. But what if you could find science-backed insights with verified citations in seconds? That’s the power of Paperpal’s new Research feature!

How to write a literature review faster with Paperpal?

Paperpal, an AI writing assistant, integrates powerful academic search capabilities within its writing platform. With the Research | Cite feature, you get 100% factual insights, with citations backed by 250M+ verified research articles, directly within your writing interface. It also allows you auto-cite references in 10,000+ styles and save relevant references in your Citation Library. By eliminating the need to switch tabs to find answers to all your research questions, Paperpal saves time and helps you stay focused on your writing.

Here’s how to use the Research feature:

Ask a question: Get started with a new document on paperpal.com. Click on the “Research | Cite” feature and type your question in plain English. Paperpal will scour over 250 million research articles, including conference papers and preprints, to provide you with accurate insights and citations.

Review and Save: Paperpal summarizes the information, while citing sources and listing relevant reads. You can quickly scan the results to identify relevant references and save these directly to your built-in citations library for later access.
Cite with Confidence: Paperpal makes it easy to incorporate relevant citations and references in 10,000+ styles into your writing, ensuring your arguments are well-supported by credible sources. This translates to a polished, well-researched literature review.

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a good literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. By combining effortless research with an easy citation process, Paperpal Research streamlines the literature review process and empowers you to write faster and with more confidence. Try Paperpal Research now and see for yourself.

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic.

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods.

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers. Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved. Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic. Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings. Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject. It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review: Introduction: Provide an overview of the topic. Define the scope and purpose of the literature review. State the research question or objective. Body: Organize the literature by themes, concepts, or chronology. Critically analyze and evaluate each source. Discuss the strengths and weaknesses of the studies. Highlight any methodological limitations or biases. Identify patterns, connections, or contradictions in the existing research. Conclusion: Summarize the key points discussed in the literature review. Highlight the research gap. Address the research question or objective stated in the introduction. Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows:

	Annotated Bibliography	Literature Review
Purpose	List of citations of books, articles, and other sources with a brief description (annotation) of each source.	Comprehensive and critical analysis of existing literature on a specific topic.
Focus	Summary and evaluation of each source, including its relevance, methodology, and key findings.	Provides an overview of the current state of knowledge on a particular subject and identifies gaps, trends, and patterns in existing literature.
Structure	Each citation is followed by a concise paragraph (annotation) that describes the source’s content, methodology, and its contribution to the topic.	The literature review is organized thematically or chronologically and involves a synthesis of the findings from different sources to build a narrative or argument.
Length	Typically 100-200 words	Length of literature review ranges from a few pages to several chapters
Independence	Each source is treated separately, with less emphasis on synthesizing the information across sources.	The writer synthesizes information from multiple sources to present a cohesive overview of the topic.

References

Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234.
Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis.
Cantero, C. (2019). How to write a literature review.  San José State University Writing Center .

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 22+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.

Experience the future of academic writing – Sign up to Paperpal and start writing for free!

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, academic integrity vs academic dishonesty: types & examples, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , the ai revolution: authors’ role in upholding academic..., the future of academia: how ai tools are..., how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide).

USC Libraries
Research Guides

Organizing Your Social Sciences Research Paper

5. The Literature Review
Purpose of Guide
Design Flaws to Avoid
Independent and Dependent Variables
Glossary of Research Terms
Reading Research Effectively
Narrowing a Topic Idea
Broadening a Topic Idea
Extending the Timeliness of a Topic Idea
Academic Writing Style
Applying Critical Thinking
Choosing a Title
Making an Outline
Paragraph Development
Research Process Video Series
Executive Summary
The C.A.R.S. Model
Background Information
The Research Problem/Question
Theoretical Framework
Citation Tracking
Content Alert Services
Evaluating Sources
Primary Sources
Secondary Sources
Tiertiary Sources
Scholarly vs. Popular Publications
Qualitative Methods
Quantitative Methods
Insiderness
Using Non-Textual Elements
Limitations of the Study
Common Grammar Mistakes
Writing Concisely
Avoiding Plagiarism
Footnotes or Endnotes?
Further Readings
Generative AI and Writing
USC Libraries Tutorials and Other Guides
Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

Give a new interpretation of old material or combine new with old interpretations,
Trace the intellectual progression of the field, including major debates,
Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

Place each work in the context of its contribution to understanding the research problem being studied.
Describe the relationship of each work to the others under consideration.
Identify new ways to interpret prior research.
Reveal any gaps that exist in the literature.
Resolve conflicts amongst seemingly contradictory previous studies.
Identify areas of prior scholarship to prevent duplication of effort.
Point the way in fulfilling a need for additional research.
Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE: Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews." Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I. Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
An explanation of how each work is similar to and how it varies from the others,
Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
Persuasiveness -- which of the author's theses are most convincing or least convincing?
Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II. Development of the Literature Review

Four Basic Stages of Writing 1. Problem formulation -- which topic or field is being examined and what are its component issues? 2. Literature search -- finding materials relevant to the subject being explored. 3. Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4. Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1. Roughly how many sources would be appropriate to include? 2. What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3. Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4. Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5. Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III. Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

Current Situation : Information necessary to understand the current topic or focus of the literature review.
Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
Standards : Description of the way in which you present your information.
Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV. Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V. Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

Sources in your literature review do not clearly relate to the research problem;
You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
Does not describe the search procedures that were used in identifying the literature to review;
Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

How are they organizing their ideas?
What methods have they used to study the problem?
What theories have been used to explain, predict, or understand their research problem?
What sources have they cited to support their conclusions?
How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research? Does it forge a new path? Or, does is merely add more of the same thing being said?
Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

<< Previous: Theoretical Framework
Next: Citation Tracking >>
Last Updated: Sep 3, 2024 1:54 PM
URL: https://libguides.usc.edu/writingguide

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

An introductory paragraph that explains what your working topic and thesis is
A forecast of key topics or texts that will appear in the review
Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
Critically Evaluate: Mention the strengths and weaknesses of your sources
Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

Summarize the key findings you have taken from the literature and emphasize their significance
Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
Qualitative versus quantitative research
Empirical versus theoretical scholarship
Divide the research by sociological, historical, or cultural sources
Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

Knowledge Base
Dissertation
What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

There are five key steps to writing a literature review:

Search for relevant literature
Evaluate sources
Identify themes, debates and gaps
Outline the structure
Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

Quick Run-through
Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

Demonstrate your familiarity with the topic and scholarly context
Develop a theoretical framework and methodology for your research
Position yourself in relation to other researchers and theorists
Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

Prevent plagiarism, run a free check.

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
Body image, self-perception, self-esteem, mental health
Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

Your university’s library catalogue
Google Scholar
Project Muse (humanities and social sciences)
Medline (life sciences and biomedicine)
EconLit (economics)
Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

What question or problem is the author addressing?
What are the key concepts and how are they defined?
What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
What are the results and conclusions of the study?
How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
Themes: what questions or concepts recur across the literature?
Debates, conflicts and contradictions: where do sources disagree?
Pivotal publications: are there any influential theories or studies that changed the direction of the field?
Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

Most research has focused on young women.
There is an increasing interest in the visual aspects of social media.
But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

Methodological

Look at what results have emerged in qualitative versus quantitative research
Discuss how the topic has been approached by empirical versus theoretical scholarship
Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
Critically evaluate: mention the strengths and weaknesses of your sources.
Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

To familiarise yourself with the current state of knowledge on your topic
To ensure that you’re not just repeating what others have already done
To identify gaps in knowledge and unresolved problems that your research can address
To develop your theoretical framework and methodology
To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 3 September 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Lydon and O'Leary libraries will be closing at 2:00pm on Wednesday, July 3rd , and will be closed on Thursday, July 4th . If you have any questions, please contact [email protected] .

University of Massachusetts Lowell
University Libraries

Survey Research: Design and Presentation

Literature Review: Definition and Context
Introduction to Survey Research Design
Planning a Thesis Proposal
Slides, Articles
Evaluating Survey Results
Related Library Databases

Literature Review for Grad Students in Education

Library Guide: Literature Review

Introduction to Literature Review

If you cannot access the above video, you can watch it here

What is a Literature Review

The purpose of an academic research paper is to express and document an original idea. Literature Review is one part of that process of writing a research paper. In a research paper, you use the literature as a starting point, a building block and as evidence of a new insight. The goal of the literature review is only to summarize and synthesize the arguments and ideas of others. You should not present your original idea.

The reading that you do as part of a literature review will answer one of two questions:

“What do we know about the subject of our study?” “Based on what we know, what conclusions can we draw about the research question?”

Notice that the conclusions to be drawn are about the research question , as opposed to a novel theory.

The types of conclusions about your research question that you want to discover are: ❖ gaps in the knowledge on a subject area ❖ questions about your topic that remain unanswered ❖ areas of disagreement in your subject area that need to be settled.

Purpose of Literature Review?

There are a number of differing descriptions of the purpose of a literature review. Primarily it is a tool for

❖ researching the history of scholarly publication on a topic

❖ becoming aware of the scholarly debate within a topic

❖ a summary or restatement of conclusions from research which has been published

❖ synthesis or recombining, comparing and contrasting, the ideas of others.

❖ evaluate sources

❖ search for gaps

A literature review provides a comprehensive overview of a topic , supporting the fundamental purpose of a research paper, which is to present a new point of view or insight on a topic. The literature review supports the new insight. It does not present or argue for it.

Structure of Literature Review

Choose a topic
Find research
Organize sources/notetaking
Evaluate Sources
Synthesize: think of this phase as a narrative .

There are various ways of organizing the literature review process- if one of these seems closer to your purpose, try it out.

Different Types of Literature Sources

<< Previous: Planning a Thesis Proposal
Next: Slides, Articles >>
Last Updated: Jan 22, 2024 2:05 PM
URL: https://libguides.uml.edu/rohland_surveys

UConn Library
Literature Review: The What, Why and How-to Guide
Introduction

Literature Review: The What, Why and How-to Guide — Introduction

Getting Started
How to Pick a Topic
Strategies to Find Sources
Evaluating Sources & Lit. Reviews
Tips for Writing Literature Reviews
Writing Literature Review: Useful Sites
Citation Resources
Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D. The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review? A literature could be written to accomplish different aims:

To develop a theory or evaluate an existing theory
To summarize the historical or existing state of a research topic
Identify a problem in a field of research

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews . Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

A research paper assigned in a course
A thesis or dissertation
A grant proposal
An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework: 10.1177/08948453211037398

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

Example : The effect of leave policies on increasing fertility: a systematic review: 10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

Example : Employment Instability and Fertility in Europe: A Meta-Analysis: 10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts . Journal of Advanced Nursing , 53 (3), 311-318.

Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis: 10.1177/05390184221113735

Literature Reviews in the Health Sciences

UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
<< Previous: Getting Started
Next: How to Pick a Topic >>
Last Updated: Sep 21, 2022 2:16 PM
URL: https://guides.lib.uconn.edu/literaturereview

Princeton University Library
Research Guides
Writing Seminars
Writing Seminar 163/164 And the Rest is Drag
What is a Literature Review?

Writing Seminar 163/164 And the Rest is Drag: What is a Literature Review?

Finding Books
Finding Articles
Specialized Databases
Academic Integrity at Princeton This link opens in a new window

Finding Examples

It may be useful to look at other reviews to learn how researchers in the field "summarize and synthesize" the literature. Most research article or dissertation in the sciences will include a section which reviews the literature. Though the section may not be labeled as such, you will quickly recognize it by the number of citations and the discussion of the literature. Another option is to look for Review Articles, which are literature reviews as a stand alone article. Here are some resources where you can find Research Articles, Review Articles and Dissertations:

Articles+ - Due to the interdisciplinary nature of gender & sexuality studies Articles+ can be a great place to start your research. Please make use of the filters on the left-hand side of the screen to help refine your searches.
Gender Studies Database & LGBT Thought and Culture - Gender Studies Database & LGBT Though and Culture have a large corpus of reviews and research articles. As with Articles+ make sure to take advantage of the filters (type of publication, publication date) to help refine your searches.
Google Scholar - Using the Cited By feature, hyperlinked below the search results, you can trace the scholarly conversation moving forward.
Dissertations @ Princeton - Provides access to many Princeton dissertations, full text is available for most published after 1996.
Purdue OWL - The Purdue OWL site provides tips and examples of literature reviews and is a great source for reviewing citation styles

*** Note about using Review Articles in your research - while they are useful in helping you to locate articles on your topic, remember that you must go to and use the original source if you intend to include a study mentioned in the review. The only time you would cite a review article is if they have made an original insight in their work that you talk about in your paper. Going to the original research paper allows you to verify the information about that study and determine whether the points made in the review are valid and accurate.

What is a literature review?

A literature review surveys scholarly articles, books and other sources relevant to a particular issue, area of research, or theory. The purpose is to offer an overview of significant literature published on a topic.

A literature review may constitute an essential chapter of a thesis or dissertation, or may be a self-contained review of writings on a subject. In either case, its purpose is to:

Place each work in the context of its contribution to the understanding of the subject under review
Describe the relationship of each work to the others under consideration
Identify new ways to interpret, and shed light on any gaps in, previous research
Resolve conflicts amongst seemingly contradictory previous studies
Identify areas of prior scholarship to prevent duplication of effort
Point the way forward for further research
Place one's original work (in the case of theses or dissertations) in the context of existing literature

A literature review can be just a simple summary of the sources, but it usually has an organizational pattern and combines both summary and synthesis. A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information. It might give a new interpretation of old material or combine new with old interpretations. Or it might trace the intellectual progression of the field, including major debates. And depending on the situation, the literature review may evaluate the sources and advise the reader on the most pertinent or relevant.

Similar to primary research, development of the literature review requires four stages:

Problem formulation—which topic or field is being examined and what are its component issues?
Literature search—finding materials relevant to the subject being explored
Data evaluation—determining which literature makes a significant contribution to the understanding of the topic
Analysis and interpretation—discussing the findings and conclusions of pertinent literature

Remember, this is a process and not necessarily a linear one. As you search and evaluate the literature, you may refine your topic or head in a different direction which will take you back to the search stage. In fact, it is useful to evaluate as you go along so you don't spend hours researching one aspect of your topic only to find yourself more interested in another.

The main focus of an academic research paper is to develop a new argument, and a research paper will contain a literature review as one of its parts. In a research paper, you use the literature as a foundation and as support for a new insight that you contribute. The focus of a literature review, however, is to summarize and synthesize the arguments and ideas of others without adding new contributions.

<< Previous: Specialized Databases
Next: Academic Integrity at Princeton >>
Subjects: Writing Program
Last Updated: Jul 16, 2024 11:39 AM
URL: https://libguides.princeton.edu/andtherestisdrag

Harvey Cushing/John Hay Whitney Medical Library

Collections
Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

Biomedical Databases
Global (Public Health) Databases
Soc. Sci., History, and Law Databases
Grey Literature
Trials Registers
Data and Statistics
Public Policy
Google Tips
Recommended Books
Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question. That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment. Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

Explains the background of research on a topic.
Demonstrates why a topic is significant to a subject area.
Discovers relationships between research studies/ideas.
Identifies major themes, concepts, and researchers on a topic.
Identifies critical gaps and points of disagreement.
Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question. The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

Make sure your research question is not too broad or too narrow. Is it manageable?
Begin writing down terms that are related to your question. These will be useful for searches later.
If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover?

This may depend on your assignment. How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search.

Where to find databases:

use the tabs on this guide
Find other databases in the Nursing Information Resources web page
More on the Medical Library web page
... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
Review the abstracts of research studies carefully. This will save you time.
Use the bibliographies and references of research studies you find to locate others.
Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
Ask your librarian for help at any time.
Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

What was the research question of the study you are reviewing? What were the authors trying to discover?
Was the research funded by a source that could influence the findings?
What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
If there are conflicting studies, why do you think that is?
How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips:

Review the abstracts carefully.
Keep careful notes so that you may track your thought processes during the research process.
Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
<< Previous: Recommended Books
Last Updated: Jun 20, 2024 9:08 AM
URL: https://guides.library.yale.edu/YSNDoctoral

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
PLoS Comput Biol
v.9(7); 2013 Jul

Ten Simple Rules for Writing a Literature Review

Marco pautasso.

1 Centre for Functional and Evolutionary Ecology (CEFE), CNRS, Montpellier, France

2 Centre for Biodiversity Synthesis and Analysis (CESAB), FRB, Aix-en-Provence, France

Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications [1] . For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively [2] . Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests [3] . Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read [4] . For such summaries to be useful, however, they need to be compiled in a professional way [5] .

When starting from scratch, reviewing the literature can require a titanic amount of work. That is why researchers who have spent their career working on a certain research issue are in a perfect position to review that literature. Some graduate schools are now offering courses in reviewing the literature, given that most research students start their project by producing an overview of what has already been done on their research issue [6] . However, it is likely that most scientists have not thought in detail about how to approach and carry out a literature review.

Reviewing the literature requires the ability to juggle multiple tasks, from finding and evaluating relevant material to synthesising information from various sources, from critical thinking to paraphrasing, evaluating, and citation skills [7] . In this contribution, I share ten simple rules I learned working on about 25 literature reviews as a PhD and postdoctoral student. Ideas and insights also come from discussions with coauthors and colleagues, as well as feedback from reviewers and editors.

Rule 1: Define a Topic and Audience

How to choose which topic to review? There are so many issues in contemporary science that you could spend a lifetime of attending conferences and reading the literature just pondering what to review. On the one hand, if you take several years to choose, several other people may have had the same idea in the meantime. On the other hand, only a well-considered topic is likely to lead to a brilliant literature review [8] . The topic must at least be:

interesting to you (ideally, you should have come across a series of recent papers related to your line of work that call for a critical summary),
an important aspect of the field (so that many readers will be interested in the review and there will be enough material to write it), and
a well-defined issue (otherwise you could potentially include thousands of publications, which would make the review unhelpful).

Ideas for potential reviews may come from papers providing lists of key research questions to be answered [9] , but also from serendipitous moments during desultory reading and discussions. In addition to choosing your topic, you should also select a target audience. In many cases, the topic (e.g., web services in computational biology) will automatically define an audience (e.g., computational biologists), but that same topic may also be of interest to neighbouring fields (e.g., computer science, biology, etc.).

Rule 2: Search and Re-search the Literature

After having chosen your topic and audience, start by checking the literature and downloading relevant papers. Five pieces of advice here:

keep track of the search items you use (so that your search can be replicated [10] ),
keep a list of papers whose pdfs you cannot access immediately (so as to retrieve them later with alternative strategies),
use a paper management system (e.g., Mendeley, Papers, Qiqqa, Sente),
define early in the process some criteria for exclusion of irrelevant papers (these criteria can then be described in the review to help define its scope), and
do not just look for research papers in the area you wish to review, but also seek previous reviews.

The chances are high that someone will already have published a literature review ( Figure 1 ), if not exactly on the issue you are planning to tackle, at least on a related topic. If there are already a few or several reviews of the literature on your issue, my advice is not to give up, but to carry on with your own literature review,

An external file that holds a picture, illustration, etc.
Object name is pcbi.1003149.g001.jpg

The bottom-right situation (many literature reviews but few research papers) is not just a theoretical situation; it applies, for example, to the study of the impacts of climate change on plant diseases, where there appear to be more literature reviews than research studies [33] .

discussing in your review the approaches, limitations, and conclusions of past reviews,
trying to find a new angle that has not been covered adequately in the previous reviews, and
incorporating new material that has inevitably accumulated since their appearance.

When searching the literature for pertinent papers and reviews, the usual rules apply:

be thorough,
use different keywords and database sources (e.g., DBLP, Google Scholar, ISI Proceedings, JSTOR Search, Medline, Scopus, Web of Science), and
look at who has cited past relevant papers and book chapters.

Rule 3: Take Notes While Reading

If you read the papers first, and only afterwards start writing the review, you will need a very good memory to remember who wrote what, and what your impressions and associations were while reading each single paper. My advice is, while reading, to start writing down interesting pieces of information, insights about how to organize the review, and thoughts on what to write. This way, by the time you have read the literature you selected, you will already have a rough draft of the review.

Of course, this draft will still need much rewriting, restructuring, and rethinking to obtain a text with a coherent argument [11] , but you will have avoided the danger posed by staring at a blank document. Be careful when taking notes to use quotation marks if you are provisionally copying verbatim from the literature. It is advisable then to reformulate such quotes with your own words in the final draft. It is important to be careful in noting the references already at this stage, so as to avoid misattributions. Using referencing software from the very beginning of your endeavour will save you time.

Rule 4: Choose the Type of Review You Wish to Write

After having taken notes while reading the literature, you will have a rough idea of the amount of material available for the review. This is probably a good time to decide whether to go for a mini- or a full review. Some journals are now favouring the publication of rather short reviews focusing on the last few years, with a limit on the number of words and citations. A mini-review is not necessarily a minor review: it may well attract more attention from busy readers, although it will inevitably simplify some issues and leave out some relevant material due to space limitations. A full review will have the advantage of more freedom to cover in detail the complexities of a particular scientific development, but may then be left in the pile of the very important papers “to be read” by readers with little time to spare for major monographs.

There is probably a continuum between mini- and full reviews. The same point applies to the dichotomy of descriptive vs. integrative reviews. While descriptive reviews focus on the methodology, findings, and interpretation of each reviewed study, integrative reviews attempt to find common ideas and concepts from the reviewed material [12] . A similar distinction exists between narrative and systematic reviews: while narrative reviews are qualitative, systematic reviews attempt to test a hypothesis based on the published evidence, which is gathered using a predefined protocol to reduce bias [13] , [14] . When systematic reviews analyse quantitative results in a quantitative way, they become meta-analyses. The choice between different review types will have to be made on a case-by-case basis, depending not just on the nature of the material found and the preferences of the target journal(s), but also on the time available to write the review and the number of coauthors [15] .

Rule 5: Keep the Review Focused, but Make It of Broad Interest

Whether your plan is to write a mini- or a full review, it is good advice to keep it focused 16 , 17 . Including material just for the sake of it can easily lead to reviews that are trying to do too many things at once. The need to keep a review focused can be problematic for interdisciplinary reviews, where the aim is to bridge the gap between fields [18] . If you are writing a review on, for example, how epidemiological approaches are used in modelling the spread of ideas, you may be inclined to include material from both parent fields, epidemiology and the study of cultural diffusion. This may be necessary to some extent, but in this case a focused review would only deal in detail with those studies at the interface between epidemiology and the spread of ideas.

While focus is an important feature of a successful review, this requirement has to be balanced with the need to make the review relevant to a broad audience. This square may be circled by discussing the wider implications of the reviewed topic for other disciplines.

Rule 6: Be Critical and Consistent

Reviewing the literature is not stamp collecting. A good review does not just summarize the literature, but discusses it critically, identifies methodological problems, and points out research gaps [19] . After having read a review of the literature, a reader should have a rough idea of:

the major achievements in the reviewed field,
the main areas of debate, and
the outstanding research questions.

It is challenging to achieve a successful review on all these fronts. A solution can be to involve a set of complementary coauthors: some people are excellent at mapping what has been achieved, some others are very good at identifying dark clouds on the horizon, and some have instead a knack at predicting where solutions are going to come from. If your journal club has exactly this sort of team, then you should definitely write a review of the literature! In addition to critical thinking, a literature review needs consistency, for example in the choice of passive vs. active voice and present vs. past tense.

Rule 7: Find a Logical Structure

Like a well-baked cake, a good review has a number of telling features: it is worth the reader's time, timely, systematic, well written, focused, and critical. It also needs a good structure. With reviews, the usual subdivision of research papers into introduction, methods, results, and discussion does not work or is rarely used. However, a general introduction of the context and, toward the end, a recapitulation of the main points covered and take-home messages make sense also in the case of reviews. For systematic reviews, there is a trend towards including information about how the literature was searched (database, keywords, time limits) [20] .

How can you organize the flow of the main body of the review so that the reader will be drawn into and guided through it? It is generally helpful to draw a conceptual scheme of the review, e.g., with mind-mapping techniques. Such diagrams can help recognize a logical way to order and link the various sections of a review [21] . This is the case not just at the writing stage, but also for readers if the diagram is included in the review as a figure. A careful selection of diagrams and figures relevant to the reviewed topic can be very helpful to structure the text too [22] .

Rule 8: Make Use of Feedback

Reviews of the literature are normally peer-reviewed in the same way as research papers, and rightly so [23] . As a rule, incorporating feedback from reviewers greatly helps improve a review draft. Having read the review with a fresh mind, reviewers may spot inaccuracies, inconsistencies, and ambiguities that had not been noticed by the writers due to rereading the typescript too many times. It is however advisable to reread the draft one more time before submission, as a last-minute correction of typos, leaps, and muddled sentences may enable the reviewers to focus on providing advice on the content rather than the form.

Feedback is vital to writing a good review, and should be sought from a variety of colleagues, so as to obtain a diversity of views on the draft. This may lead in some cases to conflicting views on the merits of the paper, and on how to improve it, but such a situation is better than the absence of feedback. A diversity of feedback perspectives on a literature review can help identify where the consensus view stands in the landscape of the current scientific understanding of an issue [24] .

Rule 9: Include Your Own Relevant Research, but Be Objective

In many cases, reviewers of the literature will have published studies relevant to the review they are writing. This could create a conflict of interest: how can reviewers report objectively on their own work [25] ? Some scientists may be overly enthusiastic about what they have published, and thus risk giving too much importance to their own findings in the review. However, bias could also occur in the other direction: some scientists may be unduly dismissive of their own achievements, so that they will tend to downplay their contribution (if any) to a field when reviewing it.

In general, a review of the literature should neither be a public relations brochure nor an exercise in competitive self-denial. If a reviewer is up to the job of producing a well-organized and methodical review, which flows well and provides a service to the readership, then it should be possible to be objective in reviewing one's own relevant findings. In reviews written by multiple authors, this may be achieved by assigning the review of the results of a coauthor to different coauthors.

Rule 10: Be Up-to-Date, but Do Not Forget Older Studies

Given the progressive acceleration in the publication of scientific papers, today's reviews of the literature need awareness not just of the overall direction and achievements of a field of inquiry, but also of the latest studies, so as not to become out-of-date before they have been published. Ideally, a literature review should not identify as a major research gap an issue that has just been addressed in a series of papers in press (the same applies, of course, to older, overlooked studies (“sleeping beauties” [26] )). This implies that literature reviewers would do well to keep an eye on electronic lists of papers in press, given that it can take months before these appear in scientific databases. Some reviews declare that they have scanned the literature up to a certain point in time, but given that peer review can be a rather lengthy process, a full search for newly appeared literature at the revision stage may be worthwhile. Assessing the contribution of papers that have just appeared is particularly challenging, because there is little perspective with which to gauge their significance and impact on further research and society.

Inevitably, new papers on the reviewed topic (including independently written literature reviews) will appear from all quarters after the review has been published, so that there may soon be the need for an updated review. But this is the nature of science [27] – [32] . I wish everybody good luck with writing a review of the literature.

Acknowledgments

Many thanks to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M. Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P. Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu for insights and discussions, and to P. Bourne, T. Matoni, and D. Smith for helpful comments on a previous draft.

Funding Statement

This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.

Research Methods

Getting Started
Literature Review Research
Research Design
Research Design By Discipline
SAGE Research Methods
Teaching with SAGE Research Methods

Literature Review

What is a Literature Review?
What is NOT a Literature Review?
Purposes of a Literature Review
Types of Literature Reviews
Literature Reviews vs. Systematic Reviews
Systematic vs. Meta-Analysis

Literature Review is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

Summarizes and analyzes previous research relevant to a topic
Includes scholarly books and articles published in academic journals
Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

Help gather ideas or information
Keep up to date in current trends and findings
Help develop new questions

A literature review is important because it:

Explains the background of research on a topic.
Demonstrates why a topic is significant to a subject area.
Helps focus your own research questions or problems
Discovers relationships between research studies/ideas.
Suggests unexplored ideas or populations
Identifies major themes, concepts, and researchers on a topic.
Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University

Keep in mind the following, a literature review is NOT:

Not an essay

Not an annotated bibliography in which you summarize each article that you have reviewed. A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper where you select resources to support one side of an issue versus another. A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

provides thorough knowledge of previous studies; introduces seminal works.
helps focus one’s own research topic.
identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
helps the researcher avoid repetition of earlier research.
suggests unexplored populations.
determines whether past studies agree or disagree; identifies controversy in the literature.
tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Theoretical Review The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015), Literature reviews vs systematic reviews. Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Diagram for "What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters"

Systematic review or meta-analysis?

A systematic review answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A meta-analysis is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

clearly stated objectives with pre-defined eligibility criteria for studies
explicit, reproducible methodology
a systematic search that attempts to identify all studies
assessment of the validity of the findings of the included studies (e.g. risk of bias)
systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis.

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review. More information on meta-analyses can be found in Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies. It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings. Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted. In that case, an integrative review is an appropriate strategy.

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

<< Previous: Getting Started
Next: Research Design >>
Last Updated: Jul 15, 2024 10:34 AM
URL: https://guides.lib.udel.edu/researchmethods

How To Write An A-Grade Literature Review

3 straightforward steps (with examples) + free template.

By: Derek Jansen (MBA) | Expert Reviewed By: Dr. Eunice Rautenbach | October 2019

Quality research is about building onto the existing work of others , “standing on the shoulders of giants”, as Newton put it. The literature review chapter of your dissertation, thesis or research project is where you synthesise this prior work and lay the theoretical foundation for your own research.

Long story short, this chapter is a pretty big deal, which is why you want to make sure you get it right . In this post, I’ll show you exactly how to write a literature review in three straightforward steps, so you can conquer this vital chapter (the smart way).

Overview: The Literature Review Process

Understanding the “ why “
Finding the relevant literature
Cataloguing and synthesising the information
Outlining & writing up your literature review
Example of a literature review

But first, the “why”…

Before we unpack how to write the literature review chapter, we’ve got to look at the why . To put it bluntly, if you don’t understand the function and purpose of the literature review process, there’s no way you can pull it off well. So, what exactly is the purpose of the literature review?

Well, there are (at least) four core functions:

For you to gain an understanding (and demonstrate this understanding) of where the research is at currently, what the key arguments and disagreements are.
For you to identify the gap(s) in the literature and then use this as justification for your own research topic.
To help you build a conceptual framework for empirical testing (if applicable to your research topic).
To inform your methodological choices and help you source tried and tested questionnaires (for interviews ) and measurement instruments (for surveys ).

Most students understand the first point but don’t give any thought to the rest. To get the most from the literature review process, you must keep all four points front of mind as you review the literature (more on this shortly), or you’ll land up with a wonky foundation.

Okay – with the why out the way, let’s move on to the how . As mentioned above, writing your literature review is a process, which I’ll break down into three steps:

Finding the most suitable literature
Understanding , distilling and organising the literature
Planning and writing up your literature review chapter

Importantly, you must complete steps one and two before you start writing up your chapter. I know it’s very tempting, but don’t try to kill two birds with one stone and write as you read. You’ll invariably end up wasting huge amounts of time re-writing and re-shaping, or you’ll just land up with a disjointed, hard-to-digest mess . Instead, you need to read first and distil the information, then plan and execute the writing.

Step 1: Find the relevant literature

Naturally, the first step in the literature review journey is to hunt down the existing research that’s relevant to your topic. While you probably already have a decent base of this from your research proposal , you need to expand on this substantially in the dissertation or thesis itself.

Essentially, you need to be looking for any existing literature that potentially helps you answer your research question (or develop it, if that’s not yet pinned down). There are numerous ways to find relevant literature, but I’ll cover my top four tactics here. I’d suggest combining all four methods to ensure that nothing slips past you:

Method 1 – Google Scholar Scrubbing

Google’s academic search engine, Google Scholar , is a great starting point as it provides a good high-level view of the relevant journal articles for whatever keyword you throw at it. Most valuably, it tells you how many times each article has been cited, which gives you an idea of how credible (or at least, popular) it is. Some articles will be free to access, while others will require an account, which brings us to the next method.

Method 2 – University Database Scrounging

Generally, universities provide students with access to an online library, which provides access to many (but not all) of the major journals.

So, if you find an article using Google Scholar that requires paid access (which is quite likely), search for that article in your university’s database – if it’s listed there, you’ll have access. Note that, generally, the search engine capabilities of these databases are poor, so make sure you search for the exact article name, or you might not find it.

Method 3 – Journal Article Snowballing

At the end of every academic journal article, you’ll find a list of references. As with any academic writing, these references are the building blocks of the article, so if the article is relevant to your topic, there’s a good chance a portion of the referenced works will be too. Do a quick scan of the titles and see what seems relevant, then search for the relevant ones in your university’s database.

Method 4 – Dissertation Scavenging

Similar to Method 3 above, you can leverage other students’ dissertations. All you have to do is skim through literature review chapters of existing dissertations related to your topic and you’ll find a gold mine of potential literature. Usually, your university will provide you with access to previous students’ dissertations, but you can also find a much larger selection in the following databases:

Open Access Theses & Dissertations
Stanford SearchWorks

Keep in mind that dissertations and theses are not as academically sound as published, peer-reviewed journal articles (because they’re written by students, not professionals), so be sure to check the credibility of any sources you find using this method. You can do this by assessing the citation count of any given article in Google Scholar. If you need help with assessing the credibility of any article, or with finding relevant research in general, you can chat with one of our Research Specialists .

Alright – with a good base of literature firmly under your belt, it’s time to move onto the next step.

Need a helping hand?

Step 2: Log, catalogue and synthesise

Once you’ve built a little treasure trove of articles, it’s time to get reading and start digesting the information – what does it all mean?

While I present steps one and two (hunting and digesting) as sequential, in reality, it’s more of a back-and-forth tango – you’ll read a little , then have an idea, spot a new citation, or a new potential variable, and then go back to searching for articles. This is perfectly natural – through the reading process, your thoughts will develop , new avenues might crop up, and directional adjustments might arise. This is, after all, one of the main purposes of the literature review process (i.e. to familiarise yourself with the current state of research in your field).

As you’re working through your treasure chest, it’s essential that you simultaneously start organising the information. There are three aspects to this:

Logging reference information
Building an organised catalogue
Distilling and synthesising the information

I’ll discuss each of these below:

2.1 – Log the reference information

As you read each article, you should add it to your reference management software. I usually recommend Mendeley for this purpose (see the Mendeley 101 video below), but you can use whichever software you’re comfortable with. Most importantly, make sure you load EVERY article you read into your reference manager, even if it doesn’t seem very relevant at the time.

2.2 – Build an organised catalogue

In the beginning, you might feel confident that you can remember who said what, where, and what their main arguments were. Trust me, you won’t. If you do a thorough review of the relevant literature (as you must!), you’re going to read many, many articles, and it’s simply impossible to remember who said what, when, and in what context . Also, without the bird’s eye view that a catalogue provides, you’ll miss connections between various articles, and have no view of how the research developed over time. Simply put, it’s essential to build your own catalogue of the literature.

I would suggest using Excel to build your catalogue, as it allows you to run filters, colour code and sort – all very useful when your list grows large (which it will). How you lay your spreadsheet out is up to you, but I’d suggest you have the following columns (at minimum):

Author, date, title – Start with three columns containing this core information. This will make it easy for you to search for titles with certain words, order research by date, or group by author.
Categories or keywords – You can either create multiple columns, one for each category/theme and then tick the relevant categories, or you can have one column with keywords.
Key arguments/points – Use this column to succinctly convey the essence of the article, the key arguments and implications thereof for your research.
Context – Note the socioeconomic context in which the research was undertaken. For example, US-based, respondents aged 25-35, lower- income, etc. This will be useful for making an argument about gaps in the research.
Methodology – Note which methodology was used and why. Also, note any issues you feel arise due to the methodology. Again, you can use this to make an argument about gaps in the research.
Quotations – Note down any quoteworthy lines you feel might be useful later.
Notes – Make notes about anything not already covered. For example, linkages to or disagreements with other theories, questions raised but unanswered, shortcomings or limitations, and so forth.

If you’d like, you can try out our free catalog template here (see screenshot below).

2.3 – Digest and synthesise

Most importantly, as you work through the literature and build your catalogue, you need to synthesise all the information in your own mind – how does it all fit together? Look for links between the various articles and try to develop a bigger picture view of the state of the research. Some important questions to ask yourself are:

What answers does the existing research provide to my own research questions ?
Which points do the researchers agree (and disagree) on?
How has the research developed over time?
Where do the gaps in the current research lie?

To help you develop a big-picture view and synthesise all the information, you might find mind mapping software such as Freemind useful. Alternatively, if you’re a fan of physical note-taking, investing in a large whiteboard might work for you.

Mind mapping is a useful way to plan your literature review.

Step 3: Outline and write it up!

Once you’re satisfied that you have digested and distilled all the relevant literature in your mind, it’s time to put pen to paper (or rather, fingers to keyboard). There are two steps here – outlining and writing:

3.1 – Draw up your outline

Having spent so much time reading, it might be tempting to just start writing up without a clear structure in mind. However, it’s critically important to decide on your structure and develop a detailed outline before you write anything. Your literature review chapter needs to present a clear, logical and an easy to follow narrative – and that requires some planning. Don’t try to wing it!

Naturally, you won’t always follow the plan to the letter, but without a detailed outline, you’re more than likely going to end up with a disjointed pile of waffle , and then you’re going to spend a far greater amount of time re-writing, hacking and patching. The adage, “measure twice, cut once” is very suitable here.

In terms of structure, the first decision you’ll have to make is whether you’ll lay out your review thematically (into themes) or chronologically (by date/period). The right choice depends on your topic, research objectives and research questions, which we discuss in this article .

Once that’s decided, you need to draw up an outline of your entire chapter in bullet point format. Try to get as detailed as possible, so that you know exactly what you’ll cover where, how each section will connect to the next, and how your entire argument will develop throughout the chapter. Also, at this stage, it’s a good idea to allocate rough word count limits for each section, so that you can identify word count problems before you’ve spent weeks or months writing!

PS – check out our free literature review chapter template…

3.2 – Get writing

With a detailed outline at your side, it’s time to start writing up (finally!). At this stage, it’s common to feel a bit of writer’s block and find yourself procrastinating under the pressure of finally having to put something on paper. To help with this, remember that the objective of the first draft is not perfection – it’s simply to get your thoughts out of your head and onto paper, after which you can refine them. The structure might change a little, the word count allocations might shift and shuffle, and you might add or remove a section – that’s all okay. Don’t worry about all this on your first draft – just get your thoughts down on paper.

Once you’ve got a full first draft (however rough it may be), step away from it for a day or two (longer if you can) and then come back at it with fresh eyes. Pay particular attention to the flow and narrative – does it fall fit together and flow from one section to another smoothly? Now’s the time to try to improve the linkage from each section to the next, tighten up the writing to be more concise, trim down word count and sand it down into a more digestible read.

Once you’ve done that, give your writing to a friend or colleague who is not a subject matter expert and ask them if they understand the overall discussion. The best way to assess this is to ask them to explain the chapter back to you. This technique will give you a strong indication of which points were clearly communicated and which weren’t. If you’re working with Grad Coach, this is a good time to have your Research Specialist review your chapter.

Finally, tighten it up and send it off to your supervisor for comment. Some might argue that you should be sending your work to your supervisor sooner than this (indeed your university might formally require this), but in my experience, supervisors are extremely short on time (and often patience), so, the more refined your chapter is, the less time they’ll waste on addressing basic issues (which you know about already) and the more time they’ll spend on valuable feedback that will increase your mark-earning potential.

Literature Review Example

In the video below, we unpack an actual literature review so that you can see how all the core components come together in reality.

Let’s Recap

In this post, we’ve covered how to research and write up a high-quality literature review chapter. Let’s do a quick recap of the key takeaways:

It is essential to understand the WHY of the literature review before you read or write anything. Make sure you understand the 4 core functions of the process.
The first step is to hunt down the relevant literature . You can do this using Google Scholar, your university database, the snowballing technique and by reviewing other dissertations and theses.
Next, you need to log all the articles in your reference manager , build your own catalogue of literature and synthesise all the research.
Following that, you need to develop a detailed outline of your entire chapter – the more detail the better. Don’t start writing without a clear outline (on paper, not in your head!)
Write up your first draft in rough form – don’t aim for perfection. Remember, done beats perfect.
Refine your second draft and get a layman’s perspective on it . Then tighten it up and submit it to your supervisor.

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

38 Comments

Thank you very much. This page is an eye opener and easy to comprehend.

This is awesome!

I wish I come across GradCoach earlier enough.

But all the same I’ll make use of this opportunity to the fullest.

Thank you for this good job.

Keep it up!

You’re welcome, Yinka. Thank you for the kind words. All the best writing your literature review.

Thank you for a very useful literature review session. Although I am doing most of the steps…it being my first masters an Mphil is a self study and one not sure you are on the right track. I have an amazing supervisor but one also knows they are super busy. So not wanting to bother on the minutae. Thank you.

You’re most welcome, Renee. Good luck with your literature review 🙂

This has been really helpful. Will make full use of it. 🙂

Thank you Gradcoach.

Really agreed. Admirable effort

thank you for this beautiful well explained recap.

Thank you so much for your guide of video and other instructions for the dissertation writing.

It is instrumental. It encouraged me to write a dissertation now.

Thank you the video was great – from someone that knows nothing thankyou

an amazing and very constructive way of presetting a topic, very useful, thanks for the effort,

It is timely

It is very good video of guidance for writing a research proposal and a dissertation. Since I have been watching and reading instructions, I have started my research proposal to write. I appreciate to Mr Jansen hugely.

I learn a lot from your videos. Very comprehensive and detailed.

Thank you for sharing your knowledge. As a research student, you learn better with your learning tips in research

I was really stuck in reading and gathering information but after watching these things are cleared thanks, it is so helpful.

Really helpful, Thank you for the effort in showing such information

This is super helpful thank you very much.

Thank you for this whole literature writing review.You have simplified the process.

I’m so glad I found GradCoach. Excellent information, Clear explanation, and Easy to follow, Many thanks Derek!

You’re welcome, Maithe. Good luck writing your literature review 🙂

Thank you Coach, you have greatly enriched and improved my knowledge

Great piece, so enriching and it is going to help me a great lot in my project and thesis, thanks so much

This is THE BEST site for ANYONE doing a masters or doctorate! Thank you for the sound advice and templates. You rock!

Thanks, Stephanie 🙂

This is mind blowing, the detailed explanation and simplicity is perfect.

I am doing two papers on my final year thesis, and I must stay I feel very confident to face both headlong after reading this article.

thank you so much.

if anyone is to get a paper done on time and in the best way possible, GRADCOACH is certainly the go to area!

This is very good video which is well explained with detailed explanation

Thank you excellent piece of work and great mentoring

Thanks, it was useful

Thank you very much. the video and the information were very helpful.

Good morning scholar. I’m delighted coming to know you even before the commencement of my dissertation which hopefully is expected in not more than six months from now. I would love to engage my study under your guidance from the beginning to the end. I love to know how to do good job

Thank you so much Derek for such useful information on writing up a good literature review. I am at a stage where I need to start writing my one. My proposal was accepted late last year but I honestly did not know where to start

Like the name of your YouTube implies you are GRAD (great,resource person, about dissertation). In short you are smart enough in coaching research work.

This is a very well thought out webpage. Very informative and a great read.

Very timely.

I appreciate.

Very comprehensive and eye opener for me as beginner in postgraduate study. Well explained and easy to understand. Appreciate and good reference in guiding me in my research journey. Thank you

Thank you. I requested to download the free literature review template, however, your website wouldn’t allow me to complete the request or complete a download. May I request that you email me the free template? Thank you.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Print Friendly

University of Texas Libraries

Literature Reviews

Steps in the literature review process.

What is a literature review?
Define your research question
Determine inclusion and exclusion criteria
Choose databases and search
Review Results
Synthesize Results
Analyze Results
Librarian Support
Artificial Intelligence (AI) Tools
You may need to some exploratory searching of the literature to get a sense of scope, to determine whether you need to narrow or broaden your focus
Identify databases that provide the most relevant sources, and identify relevant terms (controlled vocabularies) to add to your search strategy
Finalize your research question
Think about relevant dates, geographies (and languages), methods, and conflicting points of view
Conduct searches in the published literature via the identified databases
Check to see if this topic has been covered in other discipline's databases
Examine the citations of on-point articles for keywords, authors, and previous research (via references) and cited reference searching.
Save your search results in a citation management tool (such as Zotero, Mendeley or EndNote)
De-duplicate your search results
Make sure that you've found the seminal pieces -- they have been cited many times, and their work is considered foundational
Check with your professor or a librarian to make sure your search has been comprehensive
Evaluate the strengths and weaknesses of individual sources and evaluate for bias, methodologies, and thoroughness
Group your results in to an organizational structure that will support why your research needs to be done, or that provides the answer to your research question
Develop your conclusions
Are there gaps in the literature?
Where has significant research taken place, and who has done it?
Is there consensus or debate on this topic?
Which methodological approaches work best?
For example: Background, Current Practices, Critics and Proponents, Where/How this study will fit in
Organize your citations and focus on your research question and pertinent studies
Compile your bibliography

Note: The first four steps are the best points at which to contact a librarian. Your librarian can help you determine the best databases to use for your topic, assess scope, and formulate a search strategy.

Videos Tutorials about Literature Reviews

This 4.5 minute video from Academic Education Materials has a Creative Commons License and a British narrator.

Literature Reviews

What is a Literature Review?

Steps for Creating a Literature Review
Providing Evidence / Critical Analysis
Challenges when writing a Literature Review
Systematic Literature Reviews

A literature review is an academic text that surveys, synthesizes, and critically evaluates the existing literature on a specific topic. It is typically required for theses, dissertations, or long reports and serves several key purposes:

Surveying the Literature : It involves a comprehensive search and examination of relevant academic books, journal articles, and other sources related to the chosen topic.
Synthesizing Information : The literature review summarizes and organizes the information found in the literature, often identifying patterns, themes, and gaps in the current knowledge.
Critical Analysis : It critically analyzes the collected information, highlighting limitations, gaps, and areas of controversy, and suggests directions for future research.
Establishing Context : It places the current research within the broader context of the field, demonstrating how the new research builds on or diverges from previous studies.

Types of Literature Reviews

Literature reviews can take various forms, including:

Narrative Reviews : These provide a qualitative summary of the literature and are often used to give a broad overview of a topic. They may be less structured and more subjective, focusing on synthesizing the literature to support a particular viewpoint.
Systematic Reviews : These are more rigorous and structured, following a specific methodology to identify, evaluate, and synthesize all relevant studies on a particular question. They aim to minimize bias and provide a comprehensive summary of the existing evidence.
Integrative Reviews : Similar to systematic reviews, but they aim to generate new knowledge by integrating findings from different studies to develop new theories or frameworks.

Importance of Literature Reviews

Foundation for Research : They provide a solid background for new research projects, helping to justify the research question and methodology.

Identifying Gaps : Literature reviews highlight areas where knowledge is lacking, guiding future research efforts.

Building Credibility : Demonstrating familiarity with existing research enhances the credibility of the researcher and their work.

In summary, a literature review is a critical component of academic research that helps to frame the current state of knowledge, identify gaps, and provide a basis for new research.

The research, the body of current literature, and the particular objectives should all influence the structure of a literature review. It is also critical to remember that creating a literature review is an ongoing process - as one reads and analyzes the literature, one's understanding may change, which could require rearranging the literature review.

Paré, G. and Kitsiou, S. (2017) 'Methods for Literature Reviews' , in: Lau, F. and Kuziemsky, C. (eds.) Handbook of eHealth evaluation: an evidence-based approach . Victoria (BC): University of Victoria.

Perplexity AI (2024) Perplexity AI response to Kathy Neville, 31 July.

Royal Literary Fund (2024) The structure of a literature review. Available at: https://www.rlf.org.uk/resources/the-structure-of-a-literature-review/ (Accessed: 23 July 2024).

Library Services for Undergraduate Research (2024) Literature review: a definition . Available at: https://libguides.wustl.edu/our?p=302677 (Accessed: 31 July 2024).

Stack Exchange Network

Stack Exchange network consists of 183 Q&A communities including Stack Overflow , the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Literature Review versus Literature Survey. What is the difference?

I have read several articles about literature reviews. At the same time I found some guides about literature surveys . I am confused... how is a literature survey different from a literature review? What is the standard procedure to conduct a literature survey without making it a literature review?

research-process
literature-review
literature-search

2 Welcome to Academia.SE. You have a couple of different questions in your post. We encourage multiple posts for multiple questions. See our tour and help center pages. Your questions about literature surveys and reviews are closely related and match the title. You should make a second post about how to pursue research given your background, since that it unrelated. – Ben Norris Commented Dec 26, 2013 at 14:11

2 Answers 2

Reviewing the literature relevant to a given field is a standard part of doing research, as this serves to put your work into the context of the larger discipline in which you are working.

If there is an actual difference between the "literature survey" and the "literature review," it's that the latter can serve as a paper in and of itself, and is much more extensive than a literature survey, which is typically a major part of the introduction of a research paper.

The literature review as a standalone article could be compared to a "curated" overview of the literature in the field—who has done what, how do papers relate to one another, and what are the most important present and (possibly) future directions of work in such a field. Such papers can also be considerably longer than a traditional research paper, and some reviews might cite as many as a thousand references!

In comparison, the literature survey of a standard research article is usually much shorter (1-2 journal pages), and will not cite nearly as many papers (anywhere from 10 to 100, depending on the topic and the amount of relevant literature available).

2 Hi thanks for your comment. But I m still confused. I have seen survey papers are published and I have seen literature review sections in thesis. I mean aren't survey papers related to computer science are literature reviews ? – Npn Commented Jan 1, 2014 at 14:51
3 In general, "review paper" is much more commonly used than "survey paper." Maybe CS prefers "survey paper," but essentially, there's no substantial difference between them. But every paper includes some sort of synopsis of existing literature; in a review or survey paper, it's the entire paper. – aeismail Commented Jan 1, 2014 at 15:12
Thanks ,I understood that review papers should be read to do a research. – Npn Commented Jan 1, 2014 at 15:30

Well, I have written couple of survery/review articles published in prestigious journals here , here , and here and hence I think I can give you some hint on this question.

First View: One of the most important things to consider is that, these terms have been used differently in varied academic disciplines and even in some cases they are used interchangeably with negligible differences. Even in CS (my field), the way image processing scholars look at these terms may be different from networking researchers (I once experienced the comments I received from experts in image processing and realize how different they look at the works). So it might not be wrong if consider insignificant differences between these two terms.

What I describe here may be more applicable to CS. There are two different views at these terms that I describe here

Technically a feasible description around these two terms is that in survey works you should review the published papers and analyze, summarize, organize, and present findings in a novel way that can generate an original view to a certain aspect of the domain. For example, if researchers review the available research findings and conclude that electrical cars are emission-free vehicles, another researcher can review the same results and present an argument that building batteries themselves produce huge emission. The second contribution opens door for new research around emission-free production of car batteries. If we consider that survey paper is the result of literature survey, we can use the following definitions from CS journals.

According to the definition of survey paper provided by IEEE Communications Surveys & Tutorials journal (one of the best CS journals), " The term survey, as applied here, is defined to mean a survey of the literature. A survey article should provide a comprehensive review of developments in a selected area ".
In ACM Computing Survey (another prestigious CS journal), survey paper is described as “A paper that summarizes and organizes recent research results in a novel way that integrates and adds understanding to work in the field. A survey article emphasizes the classification of the existing literature, developing a perspective on the area, and evaluating trends.”
In Elsevier journal of Computer Science Review, you will see here 4 that “Critical review of the relevant literature“ is required a component of every typical survey paper.

To summarize, these two terms can be distinguished using following notes (or maybe definitions)

Literature Survey: Is the process of analyzing, summarizing, organizing, and presenting novel conclusions from the results of technical review of large number of recently published scholarly articles. The results of the literature survey can contribute to the body of knowledge when peer-reviewed and published as survey articles

Literature Review: Is the process of technically and critically reviewing published papers to extract technical and scientific metadata from the presented contents. The metadata are usually used during literature survey to technically compare different but relevant works and draw conclusions on weaknesses and strengths of the works.

Second View: The second view over literature survey and review is that in survey, researchers usually utilize the author-provided contents available in the published works to qualitatively analyze and compare them with other related works. While in the former, you should not perform qualitative analysis. Rather it should be quantitative meaning that every research work under study should be implemented and benchmarked under certain criteria. The results of this benchmarking study can be used to compare them together and criticize or appreciate the works.

So basically you can look at current literature and find which approach is dominating in your field. Hope it helps. I try to revise it if I came a cross other points or useful comments here.

3 Up vote for Comprehensive answer. – user3135645 Commented Dec 28, 2013 at 5:57
3 Nice answer (+1). I agree with you that the difference between the two terms is non-essential and preference in terminology depends mostly on the research discipline (field) and journal editors' preferences. Having said that, your distinction between the terms seems artificial, meaning that I don't see core logic that prevents applying both definitions to the opposite terms (unless I've missed some points). Also, I wanted to add that more accurate definitions should mention that literature survey or literature review is each both a process and an artifact , resulting from that process. – Aleksandr Blekh Commented May 8, 2015 at 3:50

You must log in to answer this question.

Not the answer you're looking for browse other questions tagged research-process literature-review literature-search ..

Featured on Meta
Bringing clarity to status tag usage on meta sites
Announcing a change to the data-dump process

Hot Network Questions

If a Palestinian converts to Judaism, can they get Israeli citizenship?
How to substitute URLs?
Can a quadrilateral polygon have 3 obtuse angles?
Driveway electric run using existing service poles
When was EDH key exchange introduced to SSL/TLS?
Admissibility of withdrawn confession
Why is the stall speed of an aircraft a specific speed?
quantulum abest, quo minus . .
Marie-Sklodowska-Curie actions: publish a part of the proposal just after the deadline?
Did Gandalf know he was a Maia?
A story where SETI finds a signal but it's just a boring philosophical treatise
How many ways can you make change?
What would happen if the voltage dropped below one volt and the button was not hit?
Find the radius of a circle given 2 of its coordinates and their angles.
How to find the x-coordinate of the point circled(non-differentiable points) in this trigonometric function graph?
Lore reasons for being faithless
Do eternal ordinances such as the festival of unleavened bread pose a biblical contradiction?
Is it possible to travel to USA with legal cannabis?
What is Zion's depth in the Matrix?
Do I need to validate a Genoa MET daily ticket every time?
Hardware debouncing of 3.3V high signal for an ESP32 turned on via optocoupler
Light switch that is flush or recessed (next to fridge door)
When you use the subjunctive tense for events that have happened?
Is loss of availability automatically a security incident?

How to Join
ProComm Leadership Team
Our Vision and Mission
Membership Information
Constitution
ProComm Conferences
ProComm 2024
Past Conferences
Future Conference Locations
About the Journal
Submitting a Manuscript
Search Articles on IEEE Xplore
Videos for Transactions Peer Reviewers
English, Chinese, and Spanish Abstracts
IEEE Transactions on Professional Communication Editorial Staff
Classics in Communication
The Wiley-IEEE PCS Book Series
IPCC Proceedings (IEEE Xplore)
About the Communication Resources
Interpersonal
Style and Grammar
Written Reporting
Oral Presentation
Career Development
Rhetorical Strategies
Become a ProComm Expert
ProComm Network Events
Find a ProComm Expert
About Division VI
IEEE Education Society
IEEE Industrial Electronics Society
IEEE Product Safety Engineering Society
IEEE Reliability Society
IEEE Society on Social Implications of Technology
IEEE Technology and Engineering Management Society

Guidelines for Research Reports and Integrative Lit Reviews | Samples of Research Reports | Samples of Integrative Lit Reviews | Reviewers’ Expectations

About Research Reports

The most commonly published format in the IEEE Transactions on Professional Communication, these articles report quantitative, qualitative, critical, and mixed methods studies and their results. Examples include experiments, textual analyses, content analyses, surveys, design research, interview-based studies, usability tests, and ethnographies.

About Integrative Literature Reviews

A new type of research report actively sought for the IEEE Transactions on Professional Communication, integrative literature reviews are an empirical research report that systematically collects, classifies, and analyzes a body of literature on a topic. As part of the research report, authors of integrative literature reviews describe the methodology used to search, choose and code studies, and focus on providing a critique or interpretation rather than just reporting data. Popular in other disciplines because they succinctly summarize and empirically assess all of the literature on a particular topic, these types of articles are actively recruited by the Transactions .

Guidelines:

Note : We recognize that, in our effort to focus on readers and be clear with authors, our guidelines are extensive and directive. We hope, however, this detailed guidance provides authors with the strongest possible guidance and ensures the most positive outcome possible from the peer-review process.

Formatting References

Follow the IEEE style for formatting references, which differs from the APA and MLA styles that are more widely used among professional communicators.For instructions on formatting references, see .

Formatting Text

Note specific guidelines regarding:

See the for details.

This section is intended to situate the study and explain its significance.

Open this section by:

Close this section by:

This section situates this study within the larger body of literature. Although professional communication is an interdisciplinary field and readers have eclectic interests, the one thing that connects readers of this journal is their interest in professional communication. Therefore, make sure that the literature review situates the study within the larger conversation on professional communication.

Immediately following the Literature Review heading, add a short paragraph that provides a preview of the Literature Review section. The paragraph should follow this format: SENTENCE 1: In 30 to 40 words, state the overall purpose of the section. SENTENCE 2: This section starts with

Start the Literature Review with a sub-section that has the title:Theoretical Orientation

In the Theoretical Orientation section, describe the theoretical orientation underlying the study.Some of the theory presented later should be moved to this section.

Next, explain how you selected literature to include in the review.Explicitly state which topics were chosen (and, if they were not mentioned in the discussion of the theoretical framework, explain why you chose them):

Then, theme by theme, present the relevant literature as it relates to this study.

If you are presenting a qualitative study, end the discussions of each topic with the suggestion of a relationship that will be explored in the study. Do yet present the research questions.

If you are reporting a quantitative, hypothesis testing study:

For each hypothesis tested, you might state something like, “Based on the relationship between A and B suggested in the literature, we propose this hypothesis: STATE THE HYPOTHESIS.”

Note that readers of the come from a wide variety of research traditions. So regardless of the methods employed in your study, a large group of readers will have limited experience with them. To help them follow the study—and to provide other researchers with as much information as possible so that they could replicate the study (a purpose of all research reporting)—the Methodology reporting is expanded. Some of it is to bring about transparency, some of it is to bring about greater clarity to all of the readers.

Immediately following the Methodology heading, add a short paragraph that provides a preview of the Methodology section. The paragraph should follow this format: SENTENCE 1: In 30 to 40 words, state the overall purpose of the section. SENTENCE 2: This section starts with

If you are not conducting a study in which hypotheses are being tested (which are presented at the end of the Literature Review section), repeat the research questions.

Next add a section, Choice of Research Methodology.In it, explain the choice of the research method chosen and why you chose it over other quantitative, qualitative or critical methodologies.

Next explain how the study was conducted.Note that many authors often mix methods and results in this section. Please only explain the data was collected, do not report data was collected. That will be reported in the Results section.

When describing how the data was collected, include information about each of the following though the order will vary depending on the nature of the study:

provide detailed, descriptive information about the actual participants. Save that information for the Results section. : Only explain how the data was collected—do not report any of the data that was collected. Hold that for the results section. : The use of software is not a data analysis procedure. It is merely a tool to assist with the process. However, the tool should be named in the data analysis section as the tool used.

Present the data collected and its analysis in this section.

Start the section with a short paragraph that provides a preview of the Results section. The paragraph should follow this format. SENTENCE 1: In 30 to 40 words, state the overall purpose of the section. SENTENCE 2: This section starts with

The nature of the reporting varies, depending on the nature of the study. Here are some suggestions that cover the types of research most commonly presented in the

Hypothesis testing:

Avoid providing readers with “statistics before [readers] get any narrative sense of outcomes or significance. Frontload narrative, background and support with the math. The ‘story’ of this study needs to be told in a way that makes the reading” easy.

Qualitative study:

This section closes the article by describing the broader implications of the study.This section has 3 separate sub-sections:

The sub-sections should be presented in this order.

Conclusions. Present the implications of the findings within the larger context of professional communication.Link the conclusions back to the literature cited earlier. In some research traditions, this is called the Discussion.

Limitations should openly acknowledge all of the limitations of the article.Some typical issues that need to be addressed:

Close the article with suggestions for future research that would build on this one.

Do not place an additional set of Conclusions at the end of the article.

Please write the Abstract as a structured abstract. Research has shown that these types of abstracts help readers better remember the article.The format for a structured abstract for a research article or integrative literature review is:

	Summarize your purpose and rationale (1 to 2 sentences)
	Explicitly state the research questions

Samples of Research Articles Published in the Transactions

Of a quantitative study: F. Ganier & R. Querrec, “TIP-EXE: A software tool for studying the use and understanding of procedural documents,” IEEE Transactions on Professional Communication, vol. 55, no. 2, 105-121, 2012.

Of a qualitative study: P. Bosch-Sijtsema & A. Sivunen,“Professional virtual worlds supporting computer-mediated communication, collaboration, and learning in geographically distributed contexts,” IEEE Transactions on Professional Communication, vol. 56, no. 2, 160-175, 2013.

Of a critical study: B. Kanoksilapatham, “Structure of research article introductions in three engineering subdisciplines,” IEEE Transactions on Professional Communication, vol. 55, no. 2, 294-309, 2012. Samples of Integrative Literature Reviews Published in the Transactions

J. Ramey & P. G. Rao, “The systematic literature review as a research genre,” Proceedings of the 2011 International Professional Communication Conference , 2011.

P. G. Rao & J. Ramey, “Use of mobile phones by non-literate and semi-literate people: A systematic literature review,” Proceedings of the 2011 International Professional Communication Conference , 2011.

[Note that a subscription is required to view the articles. If you do not already have a subscription, your library might.]

Reviewers’ Expectations

To learn about the criteria that reviewers consider when providing feedback on a research article or integrative literature review, click here .

Skip to main content
Skip to primary sidebar
Request Info
Search Search Site Faculty/Staff
Open Navigation Menu Menu Close Navigation Menu
Literature Review Guidelines

Making sense of what has been written on your topic.

Goals of a literature review:.

Before doing work in primary sources, historians must know what has been written on their topic. They must be familiar with theories and arguments–as well as facts–that appear in secondary sources.

Before you proceed with your research project, you too must be familiar with the literature: you do not want to waste time on theories that others have disproved and you want to take full advantage of what others have argued. You want to be able to discuss and analyze your topic.

Your literature review will demonstrate your familiarity with your topic’s secondary literature.

GUIDELINES FOR A LITERATURE REVIEW:

1) LENGTH: 8-10 pages of text for Senior Theses (485) (consult with your professor for other classes), with either footnotes or endnotes and with a works-consulted bibliography. [See also the citation guide on this site.]

2) NUMBER OF WORKS REVIEWED: Depends on the assignment, but for Senior Theses (485), at least ten is typical.

3) CHOOSING WORKS:

Your literature review must include enough works to provide evidence of both the breadth and the depth of the research on your topic or, at least, one important angle of it. The number of works necessary to do this will depend on your topic. For most topics, AT LEAST TEN works (mostly books but also significant scholarly articles) are necessary, although you will not necessarily give all of them equal treatment in your paper (e.g., some might appear in notes rather than the essay). 4) ORGANIZING/ARRANGING THE LITERATURE:

As you uncover the literature (i.e., secondary writing) on your topic, you should determine how the various pieces relate to each other. Your ability to do so will demonstrate your understanding of the evolution of literature.

You might determine that the literature makes sense when divided by time period, by methodology, by sources, by discipline, by thematic focus, by race, ethnicity, and/or gender of author, or by political ideology. This list is not exhaustive. You might also decide to subdivide categories based on other criteria. There is no “rule” on divisions—historians wrote the literature without consulting each other and without regard to the goal of fitting into a neat, obvious organization useful to students.

The key step is to FIGURE OUT the most logical, clarifying angle. Do not arbitrarily choose a categorization; use the one that the literature seems to fall into. How do you do that? For every source, you should note its thesis, date, author background, methodology, and sources. Does a pattern appear when you consider such information from each of your sources? If so, you have a possible thesis about the literature. If not, you might still have a thesis.

Consider: Are there missing elements in the literature? For example, no works published during a particular (usually fairly lengthy) time period? Or do studies appear after long neglect of a topic? Do interpretations change at some point? Does the major methodology being used change? Do interpretations vary based on sources used?

Follow these links for more help on analyzing historiography and historical perspective .

5) CONTENTS OF LITERATURE REVIEW:

The literature review is a research paper with three ingredients:

a) A brief discussion of the issue (the person, event, idea). [While this section should be brief, it needs to set up the thesis and literature that follow.] b) Your thesis about the literature c) A clear argument, using the works on topic as evidence, i.e., you discuss the sources in relation to your thesis, not as a separate topic.

These ingredients must be presented in an essay with an introduction, body, and conclusion.

6) ARGUING YOUR THESIS:

The thesis of a literature review should not only describe how the literature has evolved, but also provide a clear evaluation of that literature. You should assess the literature in terms of the quality of either individual works or categories of works. For instance, you might argue that a certain approach (e.g. social history, cultural history, or another) is better because it deals with a more complex view of the issue or because they use a wider array of source materials more effectively. You should also ensure that you integrate that evaluation throughout your argument. Doing so might include negative assessments of some works in order to reinforce your argument regarding the positive qualities of other works and approaches to the topic.

Within each group, you should provide essential information about each work: the author’s thesis, the work’s title and date, the author’s supporting arguments and major evidence.

In most cases, arranging the sources chronologically by publication date within each section makes the most sense because earlier works influenced later ones in one way or another. Reference to publication date also indicates that you are aware of this significant historiographical element.

As you discuss each work, DO NOT FORGET WHY YOU ARE DISCUSSING IT. YOU ARE PRESENTING AND SUPPORTING A THESIS ABOUT THE LITERATURE.

When discussing a particular work for the first time, you should refer to it by the author’s full name, the work’s title, and year of publication (either in parentheses after the title or worked into the sentence).

For example, “The field of slavery studies has recently been transformed by Ben Johnson’s The New Slave (2001)” and “Joe Doe argues in his 1997 study, Slavery in America, that . . . .”

Your paper should always note secondary sources’ relationship to each other, particularly in terms of your thesis about the literature (e.g., “Unlike Smith’s work, Mary Brown’s analysis reaches the conclusion that . . . .” and “Because of Anderson’s reliance on the president’s personal papers, his interpretation differs from Barry’s”). The various pieces of the literature are “related” to each other, so you need to indicate to the reader some of that relationship. (It helps the reader follow your thesis, and it convinces the reader that you know what you are talking about.)

7) DOCUMENTATION:

Each source you discuss in your paper must be documented using footnotes/endnotes and a bibliography. Providing author and title and date in the paper is not sufficient. Use correct Turabian/Chicago Manual of Style form. [See Bibliography and Footnotes/Endnotes pages.]

In addition, further supporting, but less significant, sources should be included in content foot or endnotes . (e.g., “For a similar argument to Ben Johnson’s, see John Terry, The Slave Who Was New (New York: W. W. Norton, 1985), 3-45.”)

8 ) CONCLUSION OF LITERATURE REVIEW:

Your conclusion should not only reiterate your argument (thesis), but also discuss questions that remain unanswered by the literature. What has the literature accomplished? What has not been studied? What debates need to be settled?

Additional writing guidelines

History and American Studies

About the Department
Major Requirements & Courses
What courses will I take as an History major?
What can I do with my History degree?
History 485
Methodology
Choosing a Topic
Book Reviews
Historiographic Clues
Understanding Historical Perspective
Sample Literature Review
Using Quotations
Ellipses and Brackets
Footnotes and Endnotes
Content Notes
Citation Guide
Citing Non-Print Resources
How to Annotate
Annotated Examples
Journals vs. Magazines
Understanding Plagiarism
Historians Define Plagiarism
Plagiarism Tutorial
UMW Honor System
Presentation Guidelines
Tips for Leading Seminars
Hints for Class Discussion
Speaking Center
Guidelines for a Research Paper
Library Research Plan
How to Use ILL
Database Guide
Guide to Online Research
Writing Guidelines
Recognizing Passive Voice
Introduction and Conclusion
MS Word’s Grammar and Spellcheck
Writing Center
What You Need to Know
Links to Online Primary Sources by Region
What will I learn from my American Studies major?
What courses will I take as an American Studies major?
What can I do with my American Studies degree?
American Studies 485
For Prospective Students
Honors and Award Recipients
Internships

Alumni Intros

How have History & American Studies majors built careers after earning their degrees? Learn more by clicking the image above.

Doctoral Dissertations and Projects

Evaluating the impact of the nurse-patient relationship: an integrative review.

Kathryn H. Creasey , Liberty University Follow

School of Nursing

Doctor of Nursing Practice (DNP)

Tonia Kennedy

nurse-patient relationship, patient outcomes, patient satisfaction, quality of care

Disciplines

Recommended citation.

Creasey, Kathryn H., "Evaluating The Impact of The Nurse-Patient Relationship: An Integrative Review" (2024). Doctoral Dissertations and Projects . 5933. https://digitalcommons.liberty.edu/doctoral/5933

Home care agencies are rated by patients anonymously through surveys after patients’ care episodes. The star ratings on the surveys have a direct impact on the home care agency and all of the staff working there. The star rating affects insurance companies’ reimbursement, which ultimately impacts business revenue, while the goal is to provide quality patient care. In the home care setting, nurses and patients desire continuity of care and the building of trust for competent, safe, and quality care. The literature review explored the importance of fostering the nurse-patient relationship and its impact on patient satisfaction and quality. Utilizing the PRISMA model and Melnyk's level of evidence, 25 articles were chosen for the literature review. The literature produced several themes in the desired nurse-patient relationship including trust, communication, empathy, sense of belonging, and respect. Educating nurses on the importance of these qualities of the nurse-patient relationship, and supporting them to produce the desired nurse-patient relationship, could produce more satisfied patients and better survey results.

Since August 29, 2024

Included in

Nursing Commons

Collections
Faculty Expert Gallery
Theses and Dissertations
Conferences and Events
Open Educational Resources (OER)
Explore Disciplines

Advanced Search

Notify me via email or RSS .

Faculty Authors

Submit Research
Expert Gallery Login

Student Authors

Undergraduate Submissions
Graduate Submissions
Honors Submissions

Home | About | FAQ | My Account | Accessibility Statement

Privacy Copyright

Presentation Attack Detection: A Systematic Literature Review

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options, index terms.

Computing methodologies

Artificial intelligence

Computer vision

Computer vision tasks

Recommendations

An enhanced generative adversarial network model for fingerprint presentation attack detection.

Fingerprint recognition systems have played a significant role in the field of biometric security in recent years. However, it is vulnerable to several threats which can put the biometric security system at a significant risk. Presentation attack ...

Presentation attack detection system for fake Iris: a review

The real-time solicitations of biometric systems have been extensively used for several things with the growing necessities of higher security level. There are numerous biometric traits used for person identification. In recent years, iris ...

An Enhanced Generative Adversarial Network Model for Fingerprint Presentation Attack Detection

Automatic fingerprint recognition systems (AFRS) have played a significant role in biometric security in recent years. However, it is vulnerable to several threats which can put the AFRS at substantial risk. Presentation attack or spoofing is one ...

Information

Published in.

Association for Computing Machinery

New York, NY, United States

Publication History

Check for updates, author tags.

Presentation Attack
Face Liveness
Face Anti Spoofing

Contributors

Other metrics, bibliometrics, article metrics.

0 Total Citations
0 Total Downloads
Downloads (Last 12 months) 0
Downloads (Last 6 weeks) 0

View options

View or Download as a PDF file.

View online with eReader .

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
Download citation
Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Software product line testing: a systematic literature review

Open access
Published: 02 September 2024
Volume 29 , article number 146 , ( 2024 )

Cite this article

You have full access to this open access article

Halimeh Agh ORCID: orcid.org/0000-0003-0272-9092 1 ,
Aidin Azamnouri 1 &
Stefan Wagner 1 , 2

A Software Product Line (SPL) is a software development paradigm in which a family of software products shares a set of core assets. Testing has a vital role in both single-system development and SPL development in identifying potential faults by examining the behavior of a product or products, but it is especially challenging in SPL. There have been many research contributions in the SPL testing field; therefore, assessing the current state of research and practice is necessary to understand the progress in testing practices and to identify the gap between required techniques and existing approaches. This paper aims to survey existing research on SPL testing to provide researchers and practitioners with up-to-date evidence and issues that enable further development of the field. To this end, we conducted a Systematic Literature Review (SLR) with seven research questions in which we identified and analyzed 118 studies dating from 2003 to 2022. The results indicate that the literature proposes many techniques for specific aspects (e.g., controlling cost/effort in SPL testing); however, other elements (e.g., regression testing and non-functional testing) still need to be covered by existing research. Furthermore, most approaches are evaluated by only one empirical method, most of which are academic evaluations. This may jeopardize the adoption of approaches in industry. The results of this study can help identify gaps in SPL testing since specific points of SPL Engineering still need to be addressed entirely.

Systematic Review on Software Product Line Testing

Software Regression Testing in Industrial Settings: Preliminary Findings from a Literature Review

Software Test Management to Improve Software Product Quality

Explore related subjects.

Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

Software Product Line (SPL) engineering has proven to be an efficient and effective strategy to decrease implementation costs, reduce time to market, and improve the quality of derived products (Denger and Kolb 2006 ; Northrop et al. 2007 ). SPLs and Configurable Systems (Alves Pereira et al. 2020 ) are two approaches used in software engineering to manage and create software with varying levels of customization and flexibility. While both SPLs and configurable systems share the goal of offering flexibility and customization, they differ in their core approach. SPLs primarily emphasize the systematic reuse of components, architectures, and design patterns across a range of related software products. In contrast, configurable systems are single software products designed to be adaptable, enabling users to configure them to meet their unique requirements. We decided to limit the scope on SPL to keep the review focused.

Testing is an essential part of SPL Engineering (SPLE) to identify potential faults (Pohl and Metzger 2006 ). This activity examines core assets shared among many products, product-specific parts, and the interaction among them (McGregor 2001 ). Therefore, SPL testing includes activities from the validation of initial requirements to the acceptance testing of a specific product by customers (Da Mota Silveira Neto et al. 2011 ).

As the adoption of the SPL approach by companies has grown (Weiss 2008 ), many researchers have made contributions in the SPL testing field to provide efficient and effective approaches that can satisfy specific needs of the industry (e.g., controlling the cost/effort of SPL testing). This resulted in many publications on different aspects of SPL testing. Therefore, analyzing research conducted in this field using well-known empirical methods is required to provide an overview of state-of-the-art testing practices and assess the effectiveness of the proposed approaches. To this end, Systematic Literature Reviews (SLR) and Systematic Mapping Studies (SMS) were conducted on SPL testing, but the most recent one dates back to 2014 (do Carmo Machado et al. 2014 ). While some recent research has focused on reviewing specific aspects of SPL testing, such as model-based testing of SPLs (Petry et al. 2020 ), test case prioritization for SPL (Kumar 2016 ), and combinatorial interaction testing for software product lines (Lopez-Herrejon et al. 2015 ), there has not been an SLR or SMS since 2014 that provides a comprehensive overview of the current state of SPL testing in a general context. Therefore, there is a need to update existing literature reviews (Mendes et al. 2020 ) to identify up-to-date evidence and issues that enable further development of the SPL testing field.

This paper presents an SLR to analyze interesting aspects of SPL testing that are formalized as research questions. An SLR is a rigorous and systematic method to identify, evaluate, and interpret all available research relevant to a particular research question, topic area, or phenomenon of interest (Cruzes and Dybä 2011 ). The specific aspects based on which we analyzed relevant studies are:

Characteristics of the studies focused on SPL testing.

Test levels executed throughout the SPL lifecycle.

Creating test assets by considering commonalities and variabilities.

Dealing with configuration-aware software testing.

Preserving traceability between test assets and other artifacts.

Testing non-functional requirements in an SPL.

Controlling cost/effort of SPL testing.

The SLR process was conducted from June 2022 to the end of 2022. While some of the findings derived from this SLR align with the conclusions of previous SLRs, such as the identification of existing gaps in non-functional testing for SPLs and the necessity for more robust and user-friendly testing tools, our review uncovered specific insights and unaddressed gaps in this domain that were not fully explored in prior SLRs. These include:

Variability control, referring to the disciplined management and regulation of feature variations within SPLs, alongside modeling and tracing, presents persistent challenges that require attention throughout the testing process. Variability control involves implementing strategies, such as configuration and change management, to ensure consistency and predictability in the diverse configurations of products derived from the SPL.

Novel approaches are needed for regression test selection, prioritization, and minimization, along with architecture-based regression testing, to effectively manage regression testing in SPLs.

Promoting the adoption of SPL testing practices in industrial settings necessitates addressing practical challenges, such as offering guidance for industry-specific SPL testing, and conducting industrial evaluations.

Exploring the details of test levels across the SPL lifecycle and highlighting the consequences of neglecting a particular test level can offer valuable insights for practitioners.

Studies focusing on testing SPLs rarely address traceability explicitly. Considering feature variability and configuration management, more efficient methods for modeling and representing traceability relationships are required.

The remainder of this paper is organized as follows: Sect. 2 provides background information required to understand SPL and SPL testing concepts; Sect. 3 describes how the SLR methodology has been applied; the results of the SLR are reported in Sect. 4 ; potential threats to the validity of this study and the strategies employed to mitigate them are discussed in Sect. 5 ; Sect. 6 presents a summary of the research and examines the main findings; Sect. 7 provides a survey of the related research; Sect. 8 presents concluding remarks and further research.

2 Background

This section provides a concise background on the SPL development process, variability management, and testing approaches and levels as a basis for the remainder of this article.

2.1 SPL development process

SPL is a software development paradigm to achieve economies of scale and scope by analyzing product commonalities and variabilities. As this paradigm has specific benefits such as substantial cost savings, reduction of time to market, and high productivity, many organizations, including Philips, Nokia, Cummins, and Hewlett-Packard, have adopted it (Clements and Northrop 2002 ). In SPL, a set of core assets (e.g., reference architecture and reusable components) is first developed. Specific products are then built by configuring and composing the core assets in a prescribed way with product-specific features to satisfy particular market segments (Clements and Northrop 2002 ).

The SPL development process/lifecycle can be divided into two distinct phases: Domain Engineering and Application Engineering. According to Czarnecki and Eisenecker ( 2000 , p. 20), Domain Engineering is “the activity of collecting, organizing, and storing experience in building systems or parts of systems in a particular domain in the form of reusable assets, as well as providing an adequate means for reusing these assets when building new systems.” Application Engineering is focused on deriving specific products from the core assets created during Domain Engineering; in this phase, specifics of the products are added to common parts to satisfy the particular needs of a product (Clements and Northrop 2002 ). Of these two phases, Domain Engineering demands significant resources and time. If not managed effectively, it can lead to the failure of the entire SPL (Pohl et al. 2005 , p. 9–10). Three common approaches are employed for constructing an SPL, and each of these approaches directly influences the implementation of Domain Engineering (Apel et al. 2013 ):

Proactive approaches start with a comprehensive and thorough scoping of the domain to anticipate all requirements. Subsequently, all these requirements are implemented as assets, and SPL experts typically carry out this task.

Extractive approaches follow an automated process, utilizing a set of existing product variants as input. The SPL is constructed by extracting features from these variants. Features are identified and retrieved through feature location techniques (AL-Msie’deen et al. 2013 ; Rubin and Chechik 2013 ).

Reactive approaches follow an incremental process. They take as input an existing SPL version (SPL i ) and a set of new requirements about a new product. This process results in the creation of SPL i+1 , which can produce the new product.

2.2 Variability Management in SPL

In SPL engineering, variability mechanisms are fundamental for managing diversities across products. These mechanisms, as classified by Apel et al. ( 2013 ), include annotative mechanisms, transformative mechanisms (delta-oriented), and feature-oriented mechanisms. Annotative mechanisms involve marking or annotating code to denote variability points, while transformative mechanisms, such as delta-oriented programming, describe changes required to transform one product variant into another. Feature-oriented mechanisms organize variability around features and their interactions. These variability mechanisms can be applied across all stages of the software lifecycle.

A Feature Model is commonly used in Domain Engineering to present different combinations of features. A feature model is a formal representation and graphical notation that describes the variability and relationships among features in an SPL. Feature models typically consist of features (functionalities or characteristics), feature hierarchies (representing parent-child relationships between features), and constraints (rules governing the valid combinations of features) (Pohl et al. 2005 ). Due to the presence of numerous optional features, the configuration space in feature models may exponentially increase (reaching 2 n possible configurations, where n represents the number of optional features without further constraints) (Chen and Babar 2011 ). A specific product can be derived once a complete feature configuration is established.

Although proactive approaches emphasize systematic upfront planning, modeling variabilities with feature and configuration models, and high asset reusability, reactive methods can also use feature models to represent variabilities introduced by new requirements. Configuration files or mechanisms are often used in reactive approaches to specifying how variabilities are configured in reaction to new requirements (Ghanam et al. 2010 ). Furthermore, extractive approaches may employ feature models to represent and visualize variabilities discovered in existing products. Configuration scripts or files may be used to document and manage variabilities found in the codebase (Parra et al. 2012 ).

2.3 Testing approaches and levels

There exist diverse approaches to software testing, including (Luo 2001 ; Jorgensen 2013 ):

Manual testing : Testers create and execute test cases manually to evaluate the behavior of a software application or system without using automated testing tools or scripts.

Automated Testing : Specialized testing tools and scripts are used to automate the execution of test cases and the verification of software applications or systems.

Functional testing : Focuses on verifying software functions according to specified requirements. This approach includes different levels of testing, including:

Unit Testing is conducted at the lowest level, focusing on the fundamental unit of software, referred to interchangeably as “unit,” “module,” or “component.“

Integration Testing takes place when two or more tested units are integrated into a larger structure. This testing assesses the interactions between components and evaluates the quality of the overall structure when the properties cannot be determined solely from its individual components.

System Testing aims to validate the comprehensive quality of the entire system, covering end-to-end functionality. This type of testing typically aligns with the functional and requirement specifications of the system. Additionally, it assesses non-functional quality attributes like reliability, security, and maintainability.

Acceptance Testing occurs when the developers deliver the completed system to the customers or users. The primary goal of acceptance testing is to give confidence that the system functions correctly rather than to uncover errors.

Non-functional testing : Focuses on evaluating the attributes of a software system that are not directly related to its functional behavior. Instead, non-functional testing assesses the system’s performance, reliability, scalability, security, usability, and other qualities that impact the overall user experience and the system’s ability to meet non-functional requirements.

Regression testing : Focuses on verifying that recent changes or updates to a software application have not introduced new defects or negatively affected existing functionality.

Model-based testing : Test cases are derived from models representing the software’s expected behavior. Different models can be used to generate test cases systematically, including graphical representations, mathematical models, or formal notations.

SPL testing is an essential activity in SPLE to identify potential faults (Pohl and Metzger 2006 ). Exhaustive testing in SPL is usually infeasible due to a combinatorial explosion in the number of products. Following Tevanlinna et al. ( 2004 ), Reuys et al. ( 2005 ), Käköla and Dueñas ( 2006 ), there are specific differences between single-system testing and SPL testing:

Testing is a part of both phases: Domain Engineering and Application Engineering. Domain testing is focused on testing domain artifacts (e.g., requirements, features, and source code); however, as domain artifacts include variability, completely testing the domain artifacts in domain testing is impossible. Application testing aims to detect remaining faults in a derived product mainly caused by unexpected interactions.

Test assets created in Domain Engineering (e.g., test cases, test scenarios, test results, and test data) are reused in Application Engineering to test instantiated products. To this end, test assets should be created by considering variability, which we call variant-rich test assets.

3 Systematic literature review methodology

To carry out this SLR, we followed guidelines for performing SLRs in software engineering (Kitchenham and Charters 2007 ). The steps followed in conducting this SLR are developing a review protocol, conducting the review, analyzing the results, reporting the results, and discussing the findings. The review protocol used in this SLR is explained in the following subsections. The protocol includes the formulation of research questions to achieve the objective (Sect. 3.1 ), identification of sources to extract the research papers, the search criteria and principles for selecting the relevant studies (Sect. 3.2 ), specifying a set of criteria to assess the quality of each study remained for data extraction (Sect. 3.3 ), and developing the template used for extracting data (Sect. 3.4 ).

3.1 Research questions

As previously stated, this study aims to investigate how the existing approaches deal with testing in SPL. To formulate research questions, we examined topics addressed by previous research on SPL testing (Pérez et al. 2009 ; Engström and Runeson 2011 ; Da Mota Silveira Neto et al. 2011 ; do Carmo Machado et al. 2014 ). Some of the research questions were completely reused from previous research – i.e., RQ1, RQ2, RQ3, RQ6, and RQ7 – and some of them were formulated by analyzing specific aspects that have not been investigated in detail in previous research – i.e., RQ4 and RQ5.

We reuse RQs to contrast and compare the newer research contributions with the results of previous SLRs. Yet, we identified two unique, interesting aspects: Because testing every potential configuration of an SPL is often impractical, it becomes essential to employ specific approaches for identifying valid and invalid configurations. We have examined the techniques utilized or proposed in RQ4 to address this issue. Maintaining traceability between test assets and other SPL artifacts offers substantial advantages, including enhanced reusability, impact analysis, and change management. Consequently, we designed RQ5 to investigate the techniques employed for preserving traceability. Answering these questions led to a detailed investigation of the identified studies to specify practical and research issues regarding SPL testing; therefore, the results of this study can support both industrial and academic activities. The research questions are as follows:

RQ1. How is the research on SPL testing characterized? This question intends to discuss the bibliometrics of the primary studies and the evidence available to adopt the proposed approaches.

RQ2 . What levels of tests are usually executed throughout the SPL lifecycle (i.e., Domain Engineering and Application Engineering)? There are different levels of tests, and each level is associated with a specific development phase, including unit, integration, system, and acceptance tests (Ammann and Offutt 2008 ; Jaring et al. 2008 ). This question aims to specify different test levels usually executed throughout the SPL lifecycle.

RQ3 . How are test assets created by considering commonalities and variabilities? The large number of variation points and variants in an SPL increases the number of possible testing combinations. Creating test assets for all combinations of functionality is almost impossible in practice; therefore, test assets must be created by considering commonality and variability so that they can be reused as much as possible. Furthermore, an undetected error in common core assets of an SPL can be spread to all instances depending on those assets (Pohl and Metzger 2006 ); therefore, creating test assets by considering commonalities and variabilities and testing common aspects as early as possible is essential. Answering this question led to investigating how testing approaches handle commonality and variability throughout creating/executing test assets.

RQ4 . How do SPL approaches deal with configuration-aware software testing? Testing all functionality combinations in an SPL is impossible and unnecessary since some combinations are invalid based on the constraints defined between configuration parameters. This question is intended to specify ways/techniques to detect valid and invalid combinations of configuration parameters.

RQ5 . How is the traceability between test assets and other artifacts of SPL preserved throughout the SPL lifecycle? The reusability of test assets is essential to manage the complexity of SPL testing; preserving traceability between test assets and requirements/implementation can enhance the reusability of test assets. In this sense, this question is intended to identify specific ways/techniques to achieve traceability between test assets and other artifacts throughout the SPL lifecycle.

RQ6 . How are Non-Functional Requirements (NFRs) tested in SPL? NFRs such as security, reliability, and performance are very important for SPLs, and ignoring these requirements can lead to negative results (e.g., economic loss) (Nguyen 2009 ). Therefore, systematically testing NFRs by considering commonalities and variabilities is an important aspect of SPLE. This question is intended to investigate how tests of NFRs are performed in an SPL.

RQ7 . What mechanisms have been used for controlling cost/effort of SPL testing? As SPL testing is more expensive than single-system testing, identifying specific techniques to reduce effort can provide the reader with an initial list of techniques identified by analyzing the selected studies. The specified list can be enriched regarding new publications about SPL testing.

3.2 Identification of relevant literature

The process of gathering and selecting primary studies has been performed in three stages: in the first stage, we investigated previously published literature reviews on SPL testing (Pérez et al. 2009 ; Engström and Runeson 2011 ; Da Mota Silveira Neto et al. 2011 ; do Carmo Machado et al. 2014 ) to identify the initial set of papers that have been published up to 2013. In the second stage, we updated the list of papers by searching for new papers published between 2013 and 2022; in this stage, we performed forward and backward snowballing (Webster and Watson 2002 ) to identify missing relevant papers. In the third stage, we applied inclusion and exclusion criteria to each potential primary study identified through stages one and two. Each of the three stages is explained in detail in the following subsections. We must note that we chose studies that could address at least one of the RQs while selecting primary studies. For instance, certain studies focusing on SPL verification were included because they could provide insights relevant to questions such as RQ4. An Excel file was created to be shared among the authors to document the various steps of the SLR process. This file Footnote 1 contains all the details about how we gathered and selected primary studies and how we extracted data from the chosen studies.

3.2.1 Analysis of existing reviews

By searching for existing SLRs or Systematic Mapping Studies (SMSs) on SPL testing, we found four SLRs (Engström and Runeson 2011 ; Da Mota Silveira Neto et al. 2011 , Pérez et al. 2009 ; do Carmo Machado et al. 2014 ). Engström and Runeson ( 2011 ) conducted an SMS to identify useful approaches and needs for future research; in this study, 64 papers published up to 2008 were surveyed. Da Mota Silveira Neto et al. ( 2011 ) performed an SMS to investigate state-of-the-art testing practices in SPL testing; this study analyzed a set of 45 publications from 1993 to 2009. Pérez et al. ( 2009 ) conducted an SLR to identify experience reports and initiatives carried out in the SPL testing area; in this study, 23 primary studies published up to 2009 were analyzed. do Carmo Machado et al. ( 2014 ) conducted an SLR by analyzing 49 studies published up to 2013. As the four studies followed a systematic process to gather and select the primary studies, we are confident that they covered all the primary studies in the SPL testing field published up to 2013. Using the list of primary studies in the four SLR/SMS, a set of 181 potentially relevant papers was identified, shown as stage 1.1 in Fig. 1 . By reading the titles and abstracts of the publications, papers that addressed none of the research questions were excluded. Furthermore, duplicated papers were removed, i.e., those included in more than one literature review. At the end of this stage, 97 studies were finally selected, shown as stage 1.2 in Fig. 1 .

The process of gathering and selecting primary studies

3.2.2 Gathering recent publications

In the second stage of the search process, we updated the list of primary studies by analyzing papers published between 2013 and 2022 using the following databases: IEEE Xplore, Scopus, ACM DL, Springer, and Wiley online library. To answer the stated research questions, we identified the keywords that had to be used in the search process. Variants of the terms “ software product line ”, “ software product family ”, and “ software testing ” were applied to compose the search query, as follows:

(Software Product Line OR Software Product Lines OR Software Product Family OR Software Product Families) AND (Test OR Testing) .

To evaluate the search string, we first performed a limited manual search to see whether the results of that search were among the results obtained by running the search string. The search string was adapted based on the syntax requirements of each data source used. Table 13 in Appendix A shows the forms of search strings applied to different engines and the number of papers extracted from each data source.

We obtained a set of 2,608 papers by running the search string on the search engines, shown as stage 2.1 in Fig. 1 . We excluded 161 papers as duplicates since they were retrieved from multiple search engines. Furthermore, by reading the titles and abstracts of the remaining papers, a set of 2,125 papers was identified as irrelevant since they considered testing from a single-system development perspective, not an SPL point of view. At the end of this step, we had 322 papers, shown as stage 2.2 in Fig. 1 .

In the next step, we conducted both backward and forward snowballing by examining the reference lists of all the identified papers and exploring the papers that have cited these identified papers, respectively. Following this step, 70 additional papers (20 via backward snowballing and 50 via forward snowballing) were added to the previously identified set of papers, shown as stage 2.3 in Fig. 1 . At the end of stage 2, we had a set of 392 new publications, shown in Fig. 1 as stage 2.4.

3.2.3 Primary study selection strategy

By merging the results of the two previous stages, a set of 477 papers was composed, shown as stage 3.1 in Fig. 1 . Throughout the merging process, we identified 12 papers as duplicates because the year 2013 was considered in both the SLR conducted by do Carmo Machado et al. ( 2014 ) and in the automated search stage. We defined a set of inclusion and exclusion criteria to assess each potential primary study; the criteria are presented in Table 1 . These criteria were applied to the titles and abstracts of the identified papers. The first author performed this stage. However, to reduce the researcher bias, the results of this stage were validated by the second and third authors of this paper.

At this stage, we initially applied inclusion criteria to select papers meeting all of the specified criteria for inclusion. Following this, we applied exclusion criteria to exclude papers that met one or more of the specified exclusion criteria. We included only papers evaluated via at least one empirical method, including Case study, Survey, Experiment, and Observational study (Wohlin et al. 2003 ; Sjoberg et al. 2007 ; Zhang et al. 2018 ). At the end of this stage, a set of 161 papers were selected to be subject to full-text reading, depicted in Fig. 1 as stage 3.2. The analysis results of the papers, conducted based on the inclusion and exclusion criteria, are accessible within the replication package.

3.3 Quality assessment

Quality assessment of candidate studies is recommended to be performed to ensure that studies are impartially assessed for quality (Kitchenham et al. 2016 ). To this end, we used a set of quality criteria to examine the studies, shown in Table 14 in Appendix B. These criteria were reused from the criteria proposed by Dybå and Dingsøyr ( 2008 ) and cover four main aspects related to quality, including:

Reporting : Reporting of the study’s rationale, aims, and context.

Rigor : Has a thorough and appropriate approach been applied to key research methods in the study?

Credibility : Are the findings well-presented and meaningful?

Relevance : How useful are the findings to the software industry and the research community?

We used a weighting approach to examine the candidate studies in which two optional answers with their respective score were given for each question: “Yes” = 1, and “No” = 0. Then, we assigned a quality assessment score to each study by summing up the scores given to all the questions; the total quality score for each study ranged from 0 (very poor) to 11 (very good). The two authors assessed the papers, and any discrepancies were resolved by holding sessions with all the authors.

The first three criteria shown in Table 14 in Appendix B were used as the minimum quality threshold of the review to exclude non-empirical research papers. To this end, if question 1, or both of questions 2 and 3, received a “0” response, we did not continue the quality assessment process, and the paper was excluded. The results of the quality assessment for each paper are available in the replication package. Consequently, 43 papers were excluded, and 118 were selected as primary studies, shown in Fig. 1 as stage 3.3. The list of primary studies is presented in Table 15 in Appendix C.

The analysis of the studies based on quality assessment criteria is explained in more detail in Appendix E. In summary, concerning Reporting, most of the studies performed well. While the context description could be better in some studies, approximately 82% have clear research objectives, and all studies are based on research. On average, the studies performed reasonably well in terms of Rigor. Researchers have justified the research design in almost 62% of studies to accomplish the research’s goals. A base approach has been compared with the proposed approach in around 60% of studies, with the researchers attempting to prove that the selected controls reflect a defined population. Despite these promising findings, 32% of the studies fail in rigor. According to the credibility issue, around 95% of the studies discuss the results in relation to the research questions and highlight the study’s limitations. Most studies, however, need to establish relationships between the researcher and participants and the data collection that addresses the research problem. Regarding Relevance, about 97% of studies explicitly discuss SPL testing and how it contributes to existing knowledge, identifies new areas for research, and explains how the results can be used. Nevertheless, practitioner-based guidelines are present in about 15% of cases, indicating that more practical guidance is needed to strengthen industry adoption of SPL testing.

3.4 Data extraction and analysis

Data was extracted from each of the 118 primary studies during this stage. To this end, we used a predefined extraction form that enabled us to record the full details of the studies and be specific in answering research questions. The extraction form is shown in Table 2 . The first two authors conducted the process of reading and completing the extraction form; the data were extracted and stored in a spreadsheet after reading each paper and shared with all the authors. We followed the content structuring / theme analysis approach of Mayring ( 2014 ) to analyze the data. The types of extracted data from the extraction form already provided us with a list of themes and the corresponding extracted data for these themes. This step was deductive. In the next step, we inductively created categories in the themes to summarize them. All the authors held multiple sessions to discuss the intermediate results and resolve any potential discrepancies.

In the following sections, the data extracted from the primary studies is used to answer the research questions. An overview of the primary studies is first provided in Sect. 4.1 . Then, we answer each RQ via the extracted data.

4.1 Characteristics of the studies (RQ1)

This section discusses the bibliometrics of the primary studies, the evidence available to adopt the proposed approaches, and the results of the evaluations conducted based on the quality assessment criteria.

4.1.1 Bibliometrics

In this section, we analyze annual trends and distribution per venue type of the studies selected.

Annual trend:

The distribution of the primary studies according to publication year is shown in Fig. 2 . No publication prior to 2003 focuses on SPL testing. However, after 2003, there was at least one paper per year, except for 2004. As seen in Fig. 2 , the number of published papers in this field has generally increased over time (2003–2019). This indicates that the SPL Testing field has attracted the attention of many researchers in the last few years. Furthermore, it shows increasing attention to the use of empirical methods to assess the value of proposed approaches since we only included empirically evaluated studies in our review. As we excluded some of the papers based on the quality assessment criteria, there is no primary study published in 2004 that satisfies the minimum quality threshold of the review. Furthermore, the number of papers published in some years (e.g., 2013) was actually higher than the ones presented in Fig. 2 ; however, some of those papers were excluded throughout the assessment of quality criteria. It is worth mentioning that many studies might not be made available by search engines until the time the search was performed (August 2022), and thus, we did not consider these studies in this study. We have specified these studies in the replication package. The overall trend that the number of publications increases is quantified by all entries in DBLP for each year, as shown in Fig. 2 for comparison. As we see in this figure, the trend in SPL testing is well above in several years (2014, 2016, 2017, and 2019). However, this trend has been decreasing in recent years.

Distribution of primary studies by year

Distribution per venue:

Most of the primary studies were published in conferences; of 65 conference papers, 17 papers (∼ 26%) were published in SPLC Footnote 2 , which is the most representative conference for the SPL engineering area. This indicates that SPLC is an important venue for SPL research, and most primary studies in this field are presented in SPLC. Also, 31% of studies were published in journals, 7% in symposiums, and 5% in workshops.

4.1.2 Analyzing the evidence available to adopt the proposed approaches

As reported in the title or the text of the studies, case studies, experiments, and expert surveys are the specific methods that have been used for evaluating primary studies. Most of the primary studies were evaluated by conducting an experiment (∼ 58%). It is worth mentioning that five studies applied more than one evaluation method, including case study and expert survey (Bucaioni et al. 2022 ), case study and experiment (Akbari et al. 2017 ; Fragal et al. 2019 ), experiment and expert survey (Hervieu et al. 2016 ), and case study, experiment, and expert survey (Wang et al. 2017 ). Table 3 shows the primary studies that have used each type of evaluation method.

Although the studies reported that their proposed approaches were evaluated by using the mentioned empirical methods, we need to analyze the strength of the evidence available to adopt the proposed approaches. The results of this analysis can help researchers to find new topics for empirical studies, and practitioners to assess the maturity of a proposed approach. Kitchenham and Charters ( 2007 ) classified the study design into five levels, based on the evidence presented in medical research.

Alves et al. ( 2010 ) revised the classification to be applicable in their study; the revised classification is fully applicable in our review. The following hierarchy is used in our study (from weakest to strongest):

No evidence.

Evidence obtained from demonstration or working out toy examples.

Evidence obtained from expert opinions or observations.

Evidence obtained from academic studies, e.g., controlled lab experiments.

Evidence obtained from industrial studies, e.g., causal case studies.

Evidence obtained from industrial practice.

Based on the evidence evaluation scheme explained, the results of the evaluation on how much evidence is available to adopt the proposed approaches are presented in Table 16 in Appendix D. All the studies have been evaluated by one kind of evaluation method. Academic studies (Lev4) are the most used evaluation method (60%), where open-source repositories are usually utilized to assess the proposed approaches. Following is Demonstration (Lev2) (∼ 17%). Only a small number of studies have been evaluated by using industrial systems or real data sets (∼ 16%) (Industrial studies, Lev5), or by applying the proposed methods to industrial settings and by involving industrial professionals (∼ 13%) (Industrial practice, Lev6). This analysis shows an overall low level of evidence in the SPL testing field that is in line with the results of the SLR conducted by do Carmo Machado et al. ( 2014 ).

4.2 Test levels executed throughout the SPL lifecycle (RQ2)

We divided SPL testing according to the two common phases of SPLE: Domain Engineering and Application Engineering. Based on the analysis of the studies, there are two types of testing activities that are performed during Domain Engineering: (1) developing test assets so they can be instantiated in Application Engineering, (2) applying tests to assets produced during Domain Engineering to detect faults in common core assets as soon as possible. By analyzing studies that are focused on the second activity, we identified two levels of tests usually performed in Domain Engineering; distribution of studies based on the test levels is shown in Table 4 :

Unit testing : Out of 118 studies, three studies are only focused on this level of testing (Jaring et al. 2008 ; Kim et al. 2011 , 2012 ). Jaring et al. ( 2008 ) classified test levels based on the binding time of variabilities. Based on this study, unit tests are performed before variant binding; therefore, we included this study in this classification since Application Engineering is the phase in which variabilities are bounded to derive a specific product. Kim et al. ( 2011 ) and Kim et al. ( 2012 ) proposed specific methods in which analysis on the code level is performed to generate test suits for testing common parts of an SPL in Domain Engineering.

Integration testing : Execution of integration tests in Domain Engineering are examined in the studies by Reis et al. ( 2007 ), Neto et al. ( 2010 ) and Akbari et al. ( 2017 ). Reis et al. ( 2007 ) proposed a model-based, automated technique for integration testing in Domain Engineering. In the proposed technique, integration test case scenarios are generated to support the test of interactions between the components of an integrated sub-system; placeholders are also created for necessary variable parts and all components that are not part of the integrated sub-system. Neto et al. ( 2010 ) presented a regression testing approach for SPL architectures to maintain the correctness and reliability of the architecture after modifications; as the main purpose of the approach is to verify the integration among modules and components that compose the SPL architecture, we included this study in this classification. Akbari et al. ( 2017 ) proposed a method for prioritized selection and execution of integration test cases in both Domain Engineering and Application Engineering.

Specific testing activities that are conducted in Application Engineering are: Creating specific product test assets by selecting and instantiating domain test assets, designing additional product-specific tests, and executing tests (Da Mota Silveira Neto et al. 2011 ). It is worth mentioning that some of the studies are focused on reducing the number of products that need to be tested by using specific techniques like pairwise testing (e.g., Matnei et al. 2016 ). In addition, some studies are focused on product prioritization to enhance the efficiency of SPL testing (e.g., Parejo et al. 2016 ). Once a set of configurations/products are selected/prioritized for testing, their behavior needs to be tested using a specific mechanism, e.g. executable unit tests (Parejo et al. 2016 ). Studies that are focused only on the first step (selecting/prioritizing configurations) do not usually consider a specific level of test. The testing levels usually performed in Application Engineering, as shown in Table 4 , are as follows:

Unit testing : Some of the studies considered executing unit tests in Application Engineering (Bürdek et al. 2015 ; Li et al. 2018 ; Souto and d’Amorim 2018 ; Jung et al. 2019 , 2020 ; Lochau et al. 2014 ). Bürdek et al. ( 2015 ) proposed a white-box test-suit derivation mechanism for SPLs, specifically for unit testing, in which test specifications are extended with a presence condition. A presence condition constrains the set of configurations for which a specific test case is valid; this information is used for testing configurations in Application Engineering. Li et al. ( 2018 ) investigated test cases generated for one product that are reused for another product of the SPL by applying two categories of structure-based criteria, control-flow and data-flow. Souto and d’Amorim ( 2018 ), Jung et al. ( 2019 ) and Jung et al. ( 2020 ) identify unit test cases to be selected for regression testing.

Integration testing : As shown in Table 4 , this level of testing has been considered in a greater number of studies (27 studies). Some studies have not explicitly mentioned this level of testing; however, they mentioned that the untested parts of the framework are tested during Application Engineering (Scheidemann 2006 ; Al-Dallal and Sorenson 2008 ; Jaring et al. 2008 ). Some of the studies consider the selection of integration test cases during Application Engineering (e.g., Jung et al. 2019 ).

System /Acceptance testing : This level of testing has also been considered in a greater number of studies (28 studies), as shown in Table 4 . In most studies, test models designed throughout Domain Engineering are instantiated to derive specific system test cases (e.g., Olimpiew and Gomaa 2009 ). Arrieta et al. ( 2015 ) split the lifecycle of cyber-physical systems product lines into three phases: Domain Engineering, Application Engineering, and Simulation phases. Execution of system test cases are performed in the simulation phase; however, as we classified SPL lifecycle into Domain Engineering and Application Engineering, we included this study in this category.

4.3 Creating test assets by considering commonalities and variabilities (RQ3)

Creating test assets by considering commonality and variability to enhance their reusability and to reduce the probability of undetected errors in common core assets by testing them as early as possible is essential in SPL testing. Out of 118 papers, 25 primary studies (∼ 21%) provide contributions to handle variability in a range of different manners. We conducted an exploratory analysis to identify shared characteristics among the approaches and subsequently categorized them. We identified three categories of approaches, including model-, specification-, and requirements-based approaches. The distribution of studies based on these categories is shown in Table 5 .

Model-based approaches : In model-based approaches, a set of techniques is used to design and execute tests for SPLs by leveraging formal or semi-formal models of the SPL’s variability. In the examined studies, the subsequent methods are employed to incorporate variability into test models:

Adaptation of UML models or integrating them with the feature model to produce test models including variability : In studies (Reuys et al. 2005 , 2006 ; Reis et al. 2007 ; Olimpiew and Gomaa 2009 ), activity diagrams are extended using specific mechanisms (e.g., stereotyping specific elements) to contain variabilities and then used as test models to create domain test case scenarios. Ebert et al. ( 2019 ) developed a common platform in Domain Engineering that contains all elements required for producing products. This study uses the SMArDT methodology (Drave et al. 2019 ) to elaborate each functionality defined in the platform via an extended version of activity diagram; generic test cases are then created for each functionality based on the SMArDT methodology. Reis et al. ( 2006 ) propose the ScenTED-PT technique in which the requirements and the architecture of the system are specified by UML models supplemented with performance requirements; then, they create a test model from which performance test case scenarios are derived.

Lochau et al. ( 2012a ) and Lackner et al. ( 2014 ) proposed to use the statechart modeling approach as a basis for capturing commonalities and variabilities of product implementations in an SPL; a 150% statechart model and the feature model is integrated to produce a reusable test model. The 150% statechart model is a model that contains the behavioral specification fragments of every feature without considering constraints between features, and the 100% statechart model is a specific instantiation of the 150% model by considering the dependencies and constraints (Lochau et al. 2012a ).

Using/defining different modeling notations to capture variabilities and using them to produce test assets : In this category of model-based approaches, specific modeling notations have been used or defined to create variant-rich test models. Tuglular et al. ( 2019 ) introduced Featured Event Sequence Graphs (FESGs) to explicitly capture behavioral variability in SPLs. Gebizli and Sözer ( 2016 ) used hierarchical Markov chains to model system usage; as this model captures all possible usage scenarios for a family of systems, it is considered as a reference test model. Bucaioni et al. ( 2022 ) define specific metamodels and languages to capture test variabilities, including SPL metamodel (SPLmm), Products metamodel (Pmm), Weaving metamodel (Wmm) to link features and signals in Pmm to those in SPLmm, Test case DSL (TcDSL), and Test Script generation Transformation (TsT). Fragal et al. ( 2019 ) use Featured Finite State Machines (FFSMs) to represent the abstract behavior of an SPL; in this study, the HSI method (Luo et al. 1995 ) has been extended to generate a single configurable test suite for an SPL. Luthmann et al. ( 2019a ) extended the concept of Timed Automata (TA) by feature constraints and configurable parameters to facilitate efficient verification of real-time properties for SPLs. Lochau et al. 2012b ), Lachmann et al. ( 2016 ), and Lity et al. ( 2019 ) apply the principles of delta modeling (Schaefer et al. 2010 ) to state machine test models to explicitly capture behavioral commonality and variability between product variants and then their test assets. In delta-oriented testing techniques, a product is considered as a base product and delta modules specify changes that should be applied to the base product to produce new ones (Schaefer et al. 2010 ). Beohar and Mousavi ( 2016 ) introduce the concept of Input-Output Featured Transition Systems (IOFTSs); IOFTSs are labeled transition systems with logical constraints on the presence or absence of features and are used as test models. In the work by Lochau et al. ( 2014 ), they introduced delta-oriented architecture test modeling as a means to systematically reuse common component and integration test elements across various system variants. They employed delta-oriented test artifact reuse and regression test planning to facilitate the systematic evolution of variable test elements among incrementally tested versions and/or variants of a software system.

Specification-based approaches : In these approaches, specific links are defined between different configurations of an SPL and, therefore, between test cases designed for both shared and variable components of the products. Mishra ( 2006 ) uses the process algebraic specification language CSP-CASL (Roggenbach 2006 ) to formally specify the system; then, enhancement relationships are established between the specifications of products. In this way, test cases generated for the common parts are reused between products, and new test cases are generated for the differences in the specification. Uzuncaova et al. ( 2010 ) describe properties of features as first-order logic formulas in Alloy (Jackson 2012 ); by considering a product as a base, test cases are generated for the base product using Alloy Analyzer. For each new product, the test cases from previous products are reused/refined based on the differences in the specifications.

Requirement-based approaches : In these approaches, variability is considered as early as possible so that it can be used to design test cases. In several primary studies, use case modeling is the approach used for representing requirements (Nebut et al. 2006 ; Araújo et al. 2017 ; Hajri et al. 2020 ). Nebut et al. ( 2006 ) enhance use cases with parameters and contracts used for presenting variability at the level of requirements; test-related artifacts (e.g., test objectives, test scenarios, and behavioral test patterns) are produced based on the enhanced use cases. Araújo et al. ( 2017 ) express use case specifications in a controlled natural language by considering variabilities; the specifications are then used for generating test procedures and their input and output. Hajri et al. ( 2020 ) propose to use the Product line Use case modeling Method (PUM) that supports variability modeling in use case diagrams; by using the requirement traceability mechanism, test cases for a new product are generated by reusing/adapting existing test cases or by defining new test cases.

Kang et al. ( 2015 ) propose a method called Systematic Software Product Line Test - Data (SSPLT-D) in which a set of platform test requirements are first defined throughout Domain Engineering and then platform test scenarios, platform test cases, and platform test data are created based on test requirements. Nebut et al. ( 2003 ) propose to derive a set of behavioral test patterns from the requirement model and then use them to produce product-specific test cases.

4.4 Dealing with configuration-aware software testing (RQ4)

Dealing with configuration-aware software testing, i.e., detecting valid and invalid combinations of configuration parameters, is paramount in SPL approaches because testing all combinations of SPL functionalities would be impossible and unnecessary. In our investigation, 41 out of 118 papers (∼ 35%) have addressed this. These papers have employed three methods to distinguish between valid and invalid configurations; distribution of studies based on these methods is shown in Table 6 :

Using/proposing specific approaches/algorithms/tools to produce valid configurations : Some studies utilize constraint programming, which is used for solving and modeling constraint satisfaction problems, to generate configurations that satisfy all cross-tree constraints imposed by the feature model (Hervieu et al. 2011 ; Marijan et al. 2013 ). In the same way, Kim et al. ( 2013 ) and Akbari et al. ( 2017 ) propose a constraint handling approach to produce valid configurations; as an example, an algorithm called SPLat is proposed in study (Kim et al. 2013 ) that dynamically prunes irrelevant configurations by handling constraints.

Using formal methods to check cross-tree constraints defined in feature models to check the relations between features is another way to find and produce valid configurations (Lackner et al. 2014 ; Lopez-Herrejon et al. 2014 ; Beohar and Mousavi 2016 ; Parejo et al. 2016 ; Ferrer et al. 2017 , 2021 ; Akimoto et al. 2019 ; Arrieta et al. 2019 ; Jakubovski Filho et al. 2019 ; Luthmann et al. 2019b ; Ibias et al. 2022 ). For example, Lackner et al. ( 2014 ) transform a feature model into propositional formulas so that any variable assignment that satisfies the formula is a valid configuration for the product line.

Several studies suggest the utilization of sampling algorithms and techniques to generate valid configurations (Oster et al. 2010 ; Lochau et al. 2012a ; Patel et al. 2013 ; Yu et al. 2014 ; Al-Hajjaji et al. 2016 , 2019 ; Lee and Hwang 2019 ). Combinatorial Interaction Testing (CIT) is among the commonly used sampling algorithms to exclude invalid interactions between features; in CIT, design-time decisions for variability are considered to exclude invalid interactions between features. For example, Oster et al. ( 2010 ) and Lochau et al. ( 2012a ) propose a pairwise algorithm in which dependencies and constraints between each pair of features are considered to generate all possible products that cover all valid pairs of features and their potential interactions. In a study conducted by Saini et al. ( 2022 ), they introduced a distance-based method for recognizing invalid configurations. This approach involves an initial phase where specific CIT algorithms are employed to generate real configurations. Following that, desired configurations are created, considering the availability of features in the configurations. The approach distinguishes valid from invalid configurations by applying a comparison technique to assess the differences between the actual and desired configurations.

Additionally, several studies proposed tool support for their specific approaches. They used SAT solvers to generate configurations to satisfy the feature model constraints which, in turn, reduces the configuration space to be tested (Henard et al. 2013 , 2014a , b ; Galindo et al. 2016 ; Hervieu et al. 2016 ; Souto and d’Amorim 2016; Fragal et al. 2019 ; Luthmann et al. 2019a ; Krieter et al. 2020 ; Xiang et al. 2022 ). Using or implementing a tool or toolkit to produce valid configurations has been proposed by Ensan et al. ( 2012 ), Al-Hajjaji et al. ( 2016 ), Arrieta et al. ( 2016 ), Al-Hajjaji et al. ( 2019 ) and Arrieta et al. ( 2019 ). For example, FeatureIDE has been used in studies by Al-Hajjaji et al. ( 2016 ), Arrieta et al. ( 2016 ), Al-Hajjaji et al. ( 2019 ), and Arrieta et al. ( 2019 ); this tool can generate valid configurations manually and automatically.

Runtime analysis : An alternative category of methods employs runtime analysis to differentiate intended from unintended interactions. In these methodologies, rather than relying on pre-established specifications to detect interactions, they examine runtime data to distinguish valid and invalid interactions (Reuys et al. 2006 ; Lochau et al. 2014 ; Rocha et al. 2020 ; Vidal Silva et al. 2020 ). As an example, in a study by Rocha et al. ( 2020 ), they introduced an iterative technique called VarXplorer to inspect interactions as they emerge. When provided with a test case consisting of system inputs, VarXplorer generates a Feature Interaction Graph (FIG), which is a concise representation of all pairwise interactions among features. This FIG offers a visual depiction of the features that interact, the contextual data, and the relationships between features, including cases where one feature suppresses another. By employing an iterative approach to interaction detection, developers and testers can thoroughly analyze the FIG derived from all the test cases within a test suite.

It is worth mentioning that some studies only stated that the feature model is manually analyzed to consider feature dependencies and feature grouping constraints (Olimpiew and Gomaa 2009 ; Cabral et al. 2010 ).

4.5 Preserving traceability between test assets and other artifacts (RQ5)

One of the essential factors in SPL testing is the preservation of the traceability between test assets and other artifacts throughout the SPL lifecycle. This is due to enhancing the reusability of test assets for managing the SPL testing complexity. However, in this regard, a few papers take preserving traceability into account, only 14 out of 118 (∼ 12%). We categorized these papers according to the type of the artifacts linked to test assets; distribution of studies based on this classification is shown in Table 7 :

Preserving traceability between requirements and test assets : In the majority of the studies, traceability is established between requirements, often represented using UML models (primarily use cases), and various test assets. These papers have utilized various methods, encompassing the gradually refinement of UML models into test models, direct mapping of requirements to test assets, annotation-based traceability, and the application of specific tools for automated tracing.

Reuys et al. ( 2005 ), Nebut et al. ( 2006 ), Reis et al. ( 2007 ) and Olimpiew and Gomaa ( 2009 ) use UML models to preserve the traceability between requirements and test case scenarios. In the same way, Reuys et al. ( 2006 ) enabled the traceability between different artifacts (use cases, use case scenarios, architecture scenarios, and test case scenarios) by refining use case scenarios into test case scenarios.

The manual definition of links between use cases and system test cases was mentioned by Hajri et al. ( 2020 ). Lackner et al. ( 2014 ), Gebizli and Sözer ( 2016 ) and Wang et al. ( 2017 ) created mapping relationships between variabilities modeled via the feature model and the test model to preserve traceability between requirements and test assets. Bucaioni et al. ( 2022 ) employed a metamodel to create a link between the product models and the SPL model. In this approach, the shared functionalities of the SPL are represented through a class diagram, and test cases are generated explicitly for these shared functionalities.

Adding annotations to test assets to specify their relationship with other artifacts is the approach proposed by Marijan et al. ( 2017 ); in this approach, test cases were manually annotated using tags and related to one or more test requirements; this traceability information is then used to assess the quality of test cases with respect to the requirements coverage.

In some studies, specific tools are used for automated tracing (Reis et al. 2006 ; Lochau et al. 2012a ). Reis et al. ( 2006 ) use a tool named Mercury TestDirector to preserve the traceability between requirements specification, domain performance test case scenarios, and application performance test case scenarios. Lochau et al. ( 2012a ) employed Rhapsody ATG to enable traceability between requirement models and test artifacts in an automated manner.

Preserving traceability between configurations and test assets : The solution proposed by Mishra ( 2006 ) is the definition of enhancement relationships between specifications of systems (different configurations of the SPL) and, therefore, their test cases.

It is also worth mentioning that some studies have emphasized the importance of preserving traceability between test assets and other artifacts, but they provide no mechanism in this regard (Kang et al. 2015 ; Aduni Sulaiman et al. 2019 ).

4.6 Testing non-functional requirements in SPL (RQ6)

In addition to functional requirements, there are non-functional requirements which should be tested in SPL, but only 3 out of 118 studies consider them. Various categories of NFRs have been addressed in these studies, including load testing and performance profiling (Reis et al. 2006 ), NFRs at the hardware-in-the-loop level (Arrieta et al. 2016 ), and real-time properties (Luthmann et al. 2019a ).

Reis et al. ( 2006 ) propose a technique which concentrates on load testing and performance profiling. They employ the Object Management Group’s UML Profile (Fomel 2002 ) to model performance aspects. Testing NFRs as a critical aspect of cyber-physical systems is investigated at the hardware-in-the-loop level by Arrieta et al. ( 2016 ); these requirements (e.g., the usage of memory and CPU) are modeled via the feature model and their coverage is considered by using selected test cases and the simulation process. Luthmann et al. ( 2019a ) present configurable parametric timed automata to extend the expressiveness of featured timed automata to enable efficient family-based verification of real-time properties (e.g., synchronization and execution time behaviors); the proposed modeling formalism aims to represent the behavioral variability of time-critical product lines and consider the minimum/maximum delay coverage.

4.7 Controlling cost/effort of SPL testing (RQ7)

As the cost/effort of SPL testing remains a significant concern within SPLE, numerous studies have proposed various techniques to address this issue. However, the lack of a standardized classification for these techniques has made it challenging to analyze them effectively. One notable exception is the extensive research conducted on product sampling techniques, which has been categorized into specific sub-techniques, including automatic selection, semi-automatic selection, and coverage (Varshosaz et al. 2018 ). In our analysis, we utilized these established categories to organize the diverse range of techniques proposed in the literature.

While reviewing the papers, we identified other approaches that offer potential solutions to managing the cost and effort associated with SPL testing. These approaches were categorized based on their primary contributions and grouped into distinct categories. Some of the identified approaches focus on the reuse of test assets, either from a core asset base or from previously tested products. Others provide varying degrees of automation, ranging from the implementation or utilization of specialized tools to the automation of specific testing processes, such as specification-based approaches.

Additionally, a subset of studies explored strategies for prioritizing the execution order of SPL configurations or products and the associated test cases. Another category of research aimed to minimize the size of the test suite required for testing a particular product, thereby reducing overall testing effort.

It is important to note that these techniques can often be combined. For example, test prioritization and minimization techniques can be used with sampling techniques to further optimize the cost and effort associated with SPL testing. Furthermore, the list of techniques can be enriched concerning new publications regarding SPL testing. In the rest of this section, the details of these five techniques are provided:

Reusing test assets : Based on the analysis of studies, test assets (e.g., test cases and test results) are reused in two ways, including:

Reusing test assets from a core asset base : In some studies, domain test scenarios containing variabilities are created in Domain Engineering; some of these scenarios are reused, and some of them are adapted based on the application requirements (Nebut et al. 2003 ; Reuys et al. 2005 , 2006 ; Reis et al. 2006 ). Some other studies are focused on reusing test cases by selecting them from a repository based on the application requirements (Arrieta et al. 2016 ; Wang et al. 2017 ; Lima et al. 2020 ) or by binding variabilities defined in abstract test cases based on specific criteria (e.g., coverage criteria) (Al-Dallal and Sorenson 2008 ; Olimpiew and Gomaa 2009 ; Lackner et al. 2014 ; Bürdek et al. 2015 ; Kang et al. 2015 ; Ebert et al. 2019 ; Fragal et al. 2019 ; Luthmann et al. 2019a ).

Reusing test assets between products : In some studies, test assets are reused between products by analyzing differences between the current product and previously tested products (Mishra 2006 ; Uzuncaova et al. 2010 ; Neto et al. 2010 ; Lochau et al. 2012b , 2014 ; Xu et al. 2013 ; Lachmann et al. 2015 , 2016 ; Beohar and Mousavi 2016 ; Fragal et al. 2017 ; Li et al. 2018 ; Ebert et al. 2019 ; Lity et al. 2019 ; Luthmann et al. 2019a ; Tuglular et al. 2019 ; Hajri et al. 2020 ). The technique usually used in these studies is the delta-oriented testing technique, based on regression testing principles and delta modeling concepts. By considering delta modules, test cases and test results from previously tested products can be reused and adapted for the new product.

Providing a specific level of automation : We found two ways by which the studies provide a particular level of automation:

Implementing/using a specific tool(s) : In 49 studies, authors claimed that their proposed approach is automatically performed using specific tools. However, the majority of these studies fail to provide any details regarding the specific tools employed for this purpose (e.g., Reis et al. 2006 ; Olimpiew and Gomaa 2009 ; Calvagna et al. 2013 ; Li et al. 2018 ; Safdar et al. 2021 ). Table 8 shows that only 19 of these studies have provided online access to their tools. It is worth noting that most of these tools are in the form of research prototypes. Instead of developing a novel tool tailored to their proposed approach, some studies utilize a set of pre-existing tools at various stages of their approach. For instance, in the case of Parejo et al. ( 2016 ), the Combinatorial tool and Feature Model Testing System (FMTS), as introduced by Ferreira et al. ( 2013 ), were employed to derive pairs and calculate solution fitness, respectively.

Using specific techniques that help automate the testing process. Specification-based testing was used in some studies (e.g., Mishra 2006 ) as an appropriate step in automating the testing process because of its precise nature in describing the desired properties of the system under test by using a formal language. Model-based testing is another approach that helps automate the testing process. For example, Bucaioni et al. ( 2022 ) introduced a model-based approach in which test scripts are generated from shared SPL features by model transformation.

Handling the selection of products to test : Testing all possible combinations of features is almost impossible in terms of resources and execution time (Cohen et al. 2006 ). Specific approaches have been proposed to determine a minimal set of configurations so that the correctness of the entire family can be inferred by successful verification of this set. Through our examination of the studies, we have identified diverse techniques for choosing a subset of products. These techniques have been categorized according to the provided categories for product sampling in study (Varshosaz et al. 2018 ). Distribution of studies based on these techniques are shown in Table 9 :

Automatic selection : There are two general types of automatic selection techniques, including Greedy and Meta-heuristic search:

Greedy : Greedy algorithms (Vazirani 2001 ) are focused on finding an optimal solution by an iterative approach. In the context of SPLs, the optimal solution is the configuration most close to the optimum. Specific measures are used to determine a configuration as an optimum solution in each iteration (e.g., requirements/feature coverage).

Meta-heuristic search : In this category, the problem of identifying a subset of products is considered as an optimization problem. Meta-heuristic algorithms are designed to target this problem by employing computational search within the configuration space to find an optimal subset of products (Varshosaz et al. 2018 ). Some studies have applied Evolutionary Algorithm, Random Search, and Genetic Algorithm by using an aggregation function of different objectives such as cost, number of products, number of revealed faults, pairwise coverage, and mutation score (e.g., Ensan et al. 2012 ). Some other studies propose to use multi-objective algorithms (e.g., Matnei et al. 2016 ). Hyper-heuristics are another category of approaches that have been explored in some studies to solve the problem of product sampling (e.g., Strickler et al. 2016 ). A hyper-heuristic is a methodology that can help automate configuration of heuristic algorithms and determine low-level heuristics (Jakubovski Filho et al. 2018 ). To consider user preferences throughout the selection of products as well as to make use of benefits of hyper-heuristic approaches, a preference-based hyper-heuristic approach has been proposed by Jakubovski Filho et al. ( 2018 ); this approach is an example of algorithms proposed in the field called Preference and Search-Based Software Engineering (PSBSE) (Ferreira et al. 2017b ).

Semi-automatic selection : In semi-automated selection, various factors are considered, including the desired number of generated products, the allocated sampling time, and the level of coverage, such as coverage of feature interactions. Moreover, the complete sample set or an initial set produced by other sampling techniques may serve as a starting point for the sampling process (Varshosaz et al. 2018 ). As an example, Reuling et al. ( 2015 ) propose a framework for fault-based (re-)generation of configuration samples based on feature-diagram mutation. The underlying rationale for this approach is rooted in the recognition that subsets of products generated by CIT approaches can often contain numerous redundant or less significant feature combinations. Furthermore, these approaches may overlook crucial or error-prone combinations beyond t-wise, primarily due to their black-box nature, which typically lacks consideration of domain-specific knowledge, including the fault history associated with feature combinations. The authors argue that the integration of their proposed approach with pairwise CIT sampling can potentially enhance the efficiency and effectiveness of SPL testing.

Coverage : Coverage criteria are frequently employed to ensure the quality of product sampling. One commonly utilized criterion is the coverage of feature interactions (Varshosaz et al. 2018 ). CIT techniques are focused on the interactions between different features or configuration options, as these interactions often lead to defects in software systems. These techniques are classified as greedy by Cohen et al. ( 2007 ) since they are focused on selecting a subset of configurations where each configuration covers as many uncovered combinations as possible. However, it is categorized separately in some other studies (e.g., Cmyrev and Reissing 2014 ). We also prefer to separate this category of techniques from greedy algorithms since they are specially focused on covering feature interactions. The studies that provide details of either a process or an algorithm for CIT are shown in Table 9 .

The most popular kind of CIT is pairwise testing (2-wise), a specialized notion of t-wise coverage; in t-wise testing, configurations are selected in a way that guarantees that all combinations of t features are tested. Kuhn et al. ( 2004 ) showed that 80% of bugs can be revealed by investigating interaction between two variables. Furthermore, for solving problems of large complexity, pairwise has proven to be most effective since finding inconsistencies in a model including only two features might be easier than investigating all combinations of features at once (do Carmo Machado et al. 2014 ). However, Steffens et al. ( 2012 ) revealed that the interaction of three or more features usually occurs in the SPL testing field; therefore, considering the combination of high-strength can have an important role in revealing faults. To this end, some studies claimed that their proposed approach for t-wise coverage can work with any value of t (e.g., Krieter et al. 2020 ). However, high-strength (t > 3) feature interaction can lead to a large number of valid configurations and therefore complicate the problem of t-wise coverage (Qian et al. 2018 ). Therefore, selecting a specific value for t is usually a trade-off between cost and efficiency to reveal faults.

Prioritizing configurations/test cases : Test case prioritization is focused on defining the execution order of test cases that attempts to increase their effectiveness at meeting some performance goals (Li et al. 2007 ; Catal and Mishra 2012 ). By investigating studies, we found two categories of studies in this regard:

Several studies propose approaches for prioritizing SPL configurations/products to be tested; these approaches are usually used as a complement for product selection/sampling techniques. In some of these studies, one or more objectives are defined for configuration prioritization (e.g., high failure rate and high overall requirement coverage) (Scheidemann 2006 ; Sánchez et al. 2014 ; Wang et al. 2014 ; Galindo et al. 2016 ; Parejo et al. 2016 ; Akimoto et al. 2019 ; Hierons et al. 2020 ; Pett et al. 2020 ; Ferrer et al. 2021 ); results of the evaluations conducted by Parejo et al. ( 2016 ) indicate that multi-objective prioritization typically leads to faster fault detection than mono-objective prioritization. In another category of studies, similarity between configurations with respect to feature selections is considered as a criterion for product prioritization (similarity-based prioritization) (Arrieta et al. 2015 ; Al-Hajjaji et al. 2017a , 2019 ). In these approaches, configurations are prioritized based on the dissimilarity between them so that the configuration that has the lowest value of similarity compared to previously selected configurations in terms of feature selections is chosen. Al-Hajjaji et al. ( 2017b ) propose a delta-oriented product prioritization method as similarity-based prioritization techniques do not consider all actual differences between products; in this approach, instead of comparing products to select features, delta-modeling artifacts (Clarke et al. 2010 ) are used to prioritize products.

Some studies are focused on prioritizing test cases for products. Lima et al. ( 2020 ) propose a learning-based approach is proposed to prioritize test cases in the Continuous Integration (CI) cycles of Highly Configurable Systems (HCI). Arrieta et al. ( 2015 ), Marijan et al. ( 2017 ), Markiegi et al. ( 2017 ), Arrieta et al. ( 2019 ) and Hajri et al. ( 2020 ) use specific criteria to prioritize the test cases (e.g., Fault detection capability, Test execution time, or Test case appearance frequency). In another category of studies, similarity-based approaches are proposed to prioritize test cases (e.g., Devroey et al. 2017 ; Lachmann et al. 2015 ; Lachmann et al. 2016 ). As an example, Devroey et al. ( 2017 ) propose an algorithm to generate and sort dissimilar tests to achieve good fault finding; to this end, a distance function is calculated based on the actions executed by the test case. Furthermore, to provide a good coverage of a large number of products, prioritizing test cases is also performed based on the products that may execute a test case.

Minimizing test suite : This technique is focused on minimizing the test suite size for testing a product, while preserving fault detection capability and testing coverage of the original test suite. Al-Dallal and Sorenson ( 2008 ), Stricker et al. ( 2010 ), Kim et al. ( 2012 ) and Beohar and Mousavi ( 2016 ) discuss approaches in which test cases already covered during Domain Engineering or test cases related to common parts that have already been executed in previous products are ignored. Other studies propose specific approaches to reduce redundant test executions for SPL regression testing by pruning tests that are not impacted by changes (Lachmann et al. 2016 ; Jung et al. 2019 , 2020 , 2022 ; Souto and d’Amorim 2018 ).

There are studies focused on improving test generation process to produce minimal set of test cases while achieving specific objectives (e.g., coverage and cost/time) (Patel et al. 2013 ; Wang et al. 2015 ; Gebizli and Sözer 2016 ; Akbari et al. 2017 ; Marijan et al. 2017 ; Aduni Sulaiman et al. 2019 ; Markiegi et al. 2019 ; Rocha et al. 2020 ). As an example, Akbari et al. ( 2017 ) propose a method in which features in feature model are prioritized based on the domain engineer’s decisions and the constraints that exist between features; integration test cases are then produced by considering specified priorities. Furthermore, there are approaches that are not directly focused on test suit minimization; however, they help reduce redundant execution of tests for unnecessary configurations (Kim et al. 2013 ; Souto and d’Amorim 2018 ). These approaches are focused on removing the valid configurations that are unnecessary for the execution of each test.

The distribution of studies based on the identified techniques is presented in Table 10 . As observed, the majority of the studies (∼ 62%) are focused on proposing a specific level of automation. However, many of these studies do not offer details regarding the specific tools utilized for this purpose. The second most researched category of approaches pertains to handling the selection of products to test (∼ 39%). Following this are techniques involving reusing test assets (∼ 25%), prioritizing configurations/test cases (∼ 18%), and minimizing test suite size (∼ 15%).

5 Threats to validity

In this section, we discuss the main threats associated with the validation of this study, classified according to the categorization proposed by Ampatzoglou et al. ( 2019 ). These particular threats are categorized into three categories: study selection validity, data validity, and research validity.

5.1 Study selection validity

One of the main threats to any secondary study is its inability to guarantee the inclusion of all relevant articles in the field. To mitigate this threat, a meeting involving all researchers was conducted to discuss and refine the search scope and keywords. Then, we evaluated the validity of the search string by conducting a limited manual search to see whether the results of that manual search show up in the results obtained by running the search string.

To ensure the comprehensive identification of all relevant studies in our search process, we rigorously followed the guidelines provided by Kitchenham and Charters ( 2007 ). We conducted a bibliographic search of published literature reviews in the SPL testing field. We updated the list of studies by applying a search string to multiple digital libraries and performed the backward and forward snowballing process. Therefore, we are confident that we have provided good coverage of studies in the SPL testing field.

During the primary study selection process, to minimize potential bias in applying inclusion/exclusion criteria, these criteria were clearly defined and regularly updated in our protocol. The first author applied inclusion and exclusion criteria. However, to reduce the researcher bias, the results of this stage were validated by the second and third authors of this paper.

Regarding quality assessment, we used a set of quality criteria to examine the studies. These criteria were reused from those proposed by Dybå and Dingsøyr ( 2008 ). Two researchers participated in the application of quality assessment criteria. We also conducted regular meetings to address and resolve any conflicts that arose during the process effectively.

5.2 Data validity

One of the main threats regarding data validity is data extraction bias. Subjective bias during the data extraction process has the potential to lead to an inconsistent interpretation of the extracted data by researchers. To mitigate this risk, two researchers collaborate during the data extraction phase, conducting resolution sessions to address any emerging ambiguities. Nevertheless, due to certain studies needing more explicit details on specific aspects of SPL testing, such as test levels, we had to make subjective interpretations based on information scattered throughout these studies.

Subjective bias may also lead to the misclassification of data in response to RQ3–RQ7. Since no predefined categories were available, we adopted an exploratory approach, scrutinizing the extracted data and identifying pertinent categories. To mitigate this potential issue, we introduced a structured data extraction form, conducted quality assessments on the chosen studies, and maintained ongoing discussions to ensure consistency in the data extraction process and category definitions. However, it is essential to acknowledge the potential influence of researcher bias on data extraction and presentation within this study.

5.3 Research validity

Research validity encompasses threats identified at all stages of our SLR.

We extensively searched secondary studies, as detailed in Sect. 3.2. This approach enabled us to identify research gaps, consider the scope and definition of RQs, and gain insights into the current state-of-the-art within the domain of SPL testing.

In our exploration of potential threats to the repeatability of this SLR, we acknowledge the complexity inherent in replicating research. Specifically, we highlight the concern that other researchers may not repeat the SLR with precisely the same results. To mitigate this threat, we provided the details of the SLR methodology so that other researchers can replicate the study; furthermore, we have made all the data collected during the SLR process available online. However, as subjectivity in the studies analysis is one major issue in conducting a literature review, we cannot guarantee that researchers can achieve exactly the same results.

One serious threat to the validity of the SLR is the inability to generalize the study’s results to other scenarios and application domains. We included only the studies empirically evaluated in our analysis to handle this threat. However, as most evaluations do not refer to real-world practice, the results and classifications presented in this study may not fully apply to practical settings. Moreover, our SLR intentionally focused exclusively on SPLs. This deliberate choice was made to answer specific questions tailored for SPL testing. While this focus enhances the depth of our insights into SPL testing practices, it inevitably limits the applicability of our findings to the broader context of configurable systems. The decision not to include configurable systems was strategic, considering the extensive body of literature on configurable system testing, which would have required substantial additional time and effort for comprehensive analysis.

6 Discussion

In this study, we presented a systematic review of testing approaches proposed in the SPLE field. We have investigated seven RQs:

RQ1: How is the research on SPL testing characterized?

The analysis indicates that the SPL testing field has attracted significant attention from researchers in recent years, with an increase in empirically evaluated studies. Although the overall number of publications has grown, recent years have seen a decline. Most primary studies are published in conferences, with case studies, experiments, and expert surveys being the common evaluation methods. However, the strength of evidence supporting the proposed approaches varies, with academic studies (60%) being the most common, followed by demonstrations (17%). Only a small number of studies involve industrial systems or real data sets (16%) or industrial practice (13%), indicating an overall low level of evidence in the field.

RQ2 . What levels of tests are usually executed throughout the SPL lifecycle (i.e., Domain Engineering and Application Engineering)?

In Domain Engineering, testing activities include developing test assets for later use and testing assets to detect faults early. In Application Engineering, activities involve creating specific product test assets, designing additional product-specific tests, and executing tests. Some studies focus on reducing the number of products tested or prioritizing products to enhance testing efficiency. The distribution of studies based on test levels shows that in Application Engineering, integration testing and system/acceptance testing are the most commonly reported levels. In contrast, unit testing is less frequently reported in both phases. This indicates a strong focus on higher levels of testing in the SPL testing field, particularly in the Application Engineering phase.

RQ3 . How are test assets created by considering commonalities and variabilities?

Creating test assets to address commonality and variability in SPL testing is crucial for enhancing reusability and minimizing faults in core assets. Our analysis categorized these approaches into three groups: model-based, specification-based, and requirement-based.

Model-based approaches utilize formal or semi-formal models of SPL variability to design and execute tests. Specification-based approaches define specific links between different SPL configurations and test cases. Requirement-based approaches prioritize considering variability early in test case design. The distribution of studies across these categories indicates that model-based techniques are the most commonly used in the examined studies.

RQ4 . How do SPL approaches deal with configuration-aware software testing?

Dealing with configuration-aware software testing, particularly in distinguishing valid and invalid combinations of configuration parameters, is crucial in SPL approaches. Testing all possible combinations of SPL functionalities is not only impractical but also unnecessary. The studies have employed three main methods to distinguish between valid and invalid configurations: Using/proposing specific approaches, algorithms, or tools, runtime analysis, and manual analysis. The distribution of studies across these methods indicates that the majority of the studies have either proposed specific methods or algorithms or have utilized already available tools.

RQ5 . How is the traceability between test assets and other artifacts of SPL preserved throughout the SPL lifecycle?

Preservation of traceability between test assets and other artifacts is a crucial factor in SPL testing as it enhances the reusability of test assets and manages the complexity of SPL testing. However, only a few papers consider preserving traceability throughout the SPL lifecycle. The papers are categorized based on the types of artifacts associated with test assets, focusing on preserving traceability between requirements and test assets as well as between configurations and test assets. The distribution of primary studies addressing this aspect highlights that most of the studies focus on preserving traceability between requirements and test assets.

RQ6 . How are Non-Functional Requirements (NFRs) tested in SPL?

Testing NFRs in SPLs has been rarely examined by researchers, with only three studies addressing this aspect. These studies cover various categories of NFRs, such as load testing, performance profiling, NFRs at the hardware-in-the-loop level, and real-time properties.

RQ7 . What mechanisms have been used for controlling cost/effort of SPL testing?

Various techniques have been proposed to manage the cost and effort associated with SPL testing. However, the lack of a standardized classification for these techniques has made their analysis challenging. Notably, research on product sampling techniques has been extensively categorized into sub-techniques such as automatic selection, semi-automatic selection, and coverage. Beyond sampling techniques, other approaches have emerged, categorized based on their primary contributions, including reusing test assets, providing different levels of automation, handling product selection for testing, prioritizing configurations/test cases, and minimizing the test suite size.

These techniques are often combinable, as seen in the use of test prioritization and minimization techniques alongside sampling techniques to optimize testing cost and effort further. Moreover, the list of techniques continues to evolve with new publications on SPL testing. The distribution of studies reveals that the majority focus on proposing a specific level of automation (∼ 62%). However, many studies lack details on the specific tools used for this purpose. The second most researched category involves handling the selection of products to test (∼ 39%). Additionally, techniques related to reusing test assets (∼ 25%), prioritizing configurations/test cases (∼ 18%), and minimizing test suite size (∼ 15%) are also explored.

We only included studies empirically evaluated in our analysis. In this discussion, we emphasize the maturity of evaluations conducted in these studies, highlight the contributions of the studies in addressing the research questions, present the main findings, and propose research directions to address identified gaps. It is important to note that our SLR intentionally focused exclusively on SPLs. We deliberately excluded the broader context of configurable systems from our analysis to have a clear focus for our article. Therefore, all the findings and research gaps reported in this section are based on our analysis within the SPL testing area. We acknowledge that this might lead to missing synergies with contributions from the broader field of configurable systems. Still, we hope this SLR can be the basis for exploring these aspects in future work.

6.1 Overview of evaluation maturity and studies’ contributions

Proposed approaches have been evaluated using three types of evaluation methods, including case studies, experiments, and expert surveys. However, there is variation in the scope and type of SPLs employed in these evaluations. Different types of SPLs have been employed in the evaluations, representing diverse application domains, such as embedded systems (e.g., automotive and medical systems), web-based systems, banking systems, and smartphone and vendor machine SPLs. We categorized the scope of applications employed in the evaluations into three main groups: Industrial systems with real data sets, SPLs sourced from online repositories (e.g., SPLOT repository) or extracted from existing sources, and the development of a demonstrator. It is important to note that some studies utilized more than one category of applications, for instance, both industrial SPLs and SPLs available online. Approximately 60% of the studies (71 studies) conducted evaluations using SPLs available online or derived from prior research. Around 17% (20 studies) involved the development of a demonstrator for assessing the proposed approach. Only 29% (34 studies) utilized an industrial-scale SPL (Industrial study or Industrial practice) for evaluating their approach. This issue may jeopardize the adoption of the proposed approaches in industry; therefore, proposed approaches for SPL testing need to improve from their evaluation perspective.

Discussing threats to validity is crucial in research since it helps researchers and readers understand the limitations and potential challenges associated with the study. However, an analysis of the included studies reveals that only 32 primary studies (∼ 27%) extensively discussed threats to validity. In approximately 42 studies (∼ 36%), the examination of threats to validity was brief. Notably, 44 studies (∼ 37%) entirely neglected to address this crucial aspect.

Another aspect that is worth analyzing is the distribution of the studies based on their contribution to the research questions. Figure 3 represents the frequencies of studies according to the research questions addressed by them. It should be mentioned that some studies covered more than one topic; therefore, the total amount shown in Fig. 3 exceeds the total number of studies selected for final analysis. As seen in Fig. 3 , most studies address the questions RQ7 (Controlling cost/effort of SPL testing) and RQ2 (Test levels in SPL testing). Moreover, there is notable research interest in the area of configuration-aware testing (RQ4), followed by a substantial focus on variability-aware creation of test assets (RQ3). However, some aspects of SPL testing have rarely been considered and, therefore, need new solutions, including RQ5 (Traceability between test assets and other artifacts) and RQ6 (Non-functional testing).

Distribution of studies by the contribution to the research questions

6.2 Main findings

We analyzed the data based on the content structuring/theme analysis approach of Mayring ( 2014 ). Initially, the data extracted from the extraction form provided us with a list of key challenges and sub-themes. In the next step, we inductively created categories within the themes to summarize them (analytical themes). The results of this analysis are shown in Table 11 . In the rest of this section, we present various gaps and concerns that necessitate further exploration and attention from both researchers and practitioners:

Variability management : Effective variability management in SPLs is crucial, yet it introduces complexities that can pose challenges to testing (Sect. 4.3 ). One facet that needs further exploration is the challenges associated with variability control. It demands a more in-depth investigation to identify and analyze challenges arising from the diverse features and configurations inherent in SPLs. These challenges encompass the complexities introduced by numerous potential combinations and the possibility of unforeseen interactions among variable elements. While this aspect has been previously examined, the key concern lies in the applicability of the proposed solutions and approaches in real-world scenarios. For example, one of the most investigated solutions involves selecting a subset of products for testing. However, the potential for unseen interactions between features in new products to result in faults raises doubts. Furthermore, many of the proposed approaches have only been evaluated at a proof-of-concept level, necessitating a more in-depth investigation into their suitability for industrial SPL applications.

Another crucial aspect involves examining variability modeling. This includes an analysis of the current state of variability modeling in SPL testing and an exploration of opportunities to enhance modeling techniques to address testing challenges. While model-based approaches, commonly used to create variant-rich test assets, have shown promise in SPL testing, there is still room for improvement in automating the generation of test cases and ensuring comprehensive coverage based on variability models. Utilizing model-based approaches can automate the process of transforming high-level test assets (e.g., test scenarios) and generating low-level test assets (e.g., test cases and test data).

Non-functional testing : Despite the fact that functional testing of SPLs has been extensively investigated, non-functional testing aspects need greater focus and specific methodologies (Sect. 4.6 ). This particular gap has already been acknowledged in previous literature reviews. Non-functional requirements encompass diverse dimensions, including but not limited to performance, security, usability, and scalability. While some studies have explored aspects such as real-time behaviors and performance, there remains a need for further research to comprehensively address diverse facets within this domain. Moreover, the inherent nature of non-functional requirements significantly shapes testing strategies. Considering their distinct characteristics and evaluation criteria, it is crucial to investigate how distinct testing approaches are essential for various aspects like performance testing, security testing, and usability testing.

Non-functional testing, particularly in critical areas such as performance and security, poses challenges due to its resource-intensive nature. Investigating the challenges associated with acquiring and allocating resources for thorough non-functional testing throughout the SPL lifecycle is crucial for effective quality assurance.

The complexities of seamlessly integrating non-functional testing with functional testing necessitate further exploration. Examining how the interplay between these two testing dimensions influences the overall quality assurance process will contribute valuable insights to the field.

Tool support : Given the substantial testing effort required for SPLs, the availability of tools specifically designed for SPL testing is crucial (Sect. 4.7 ). The analysis of the studies with respect to automation provided by the tools indicates that most of the tool implementations are proof-of-concept prototypes developed for validating the proposed approach. Therefore, developing more robust and user-friendly tools can significantly help practitioners in their testing efforts. This particular challenge has previously been discussed in prior literature reviews.

Some specific areas need further exploration. Evaluating the effectiveness and efficiency of existing SPL testing tools explores capabilities, limitations, and areas for improvement in tools designed for various testing activities within the SPL lifecycle. Analyzing how well testing tools adapt to changes in SPL configurations includes investigating their ability to accommodate evolving feature sets, configurations, and architectural variations, ensuring continued effectiveness. Assessing the user experience and usability of SPL testing tools explores how user-friendly and accessible tools are for practitioners involved in SPL testing, considering factors such as ease of use, learning curve, and user satisfaction.

Regression testing : Effectively handling regression testing in SPLs, where modifications to one product can affect others, presents an intricate challenge (Sect. 4.7 ). Regression test selection/prioritization/minimization and architecture-based regression testing are potential points for future research. Test case selection is focused on choosing a set of relevant test cases to test the modified version of the system, and the aim of test minimization is to remove the redundant/irrelevant test cases from the existing test suit. Test case prioritization aims at ordering and ranking test cases based on specific criteria such as importance and likelihood of failure. All these techniques aim to reduce the cost/effort of SPL testing after applying any change to products or the SPL architecture.

An important aspect is analyzing how changes and evolutions in the SPL architecture impact regression testing strategies. This investigation includes understanding the challenges of maintaining test suites across evolving SPL configurations and the need for adaptive regression testing approaches.

Additionally, exploring the benefits and challenges of implementing automated regression testing within the SPL context is crucial. This requires an analysis of efficiency gains, potential pitfalls, and strategies to optimize the effectiveness of automated regression testing in SPL scenarios.

Moreover, investigating challenges related to maintaining traceability between evolving codebase versions and regression test suites is critical. This requires exploring strategies to preserve traceability links, ensuring that regression testing aligns with the dynamic nature of SPL development.

Industrial evaluations : Encouraging the adoption of SPL testing practices in industrial settings requires addressing practical challenges (Sect. 3.3 and 4.1 ). This includes offering guidance tailored for industry-specific SPL testing and conducting industrial evaluations.

To enhance the industry adoption of SPL approaches, offering practical insights and recommendations is essential. This involves providing tailored guidance to help organizations navigate the unique challenges and requirements of adopting SPL testing methods in their specific industry domains. Additionally, there is a need to move beyond proof-of-concept evaluations and conduct practical assessments to verify the feasibility, scalability, and effectiveness of proposed SPL testing methods in diverse industrial contexts.

Test levels throughout the SPL lifecycle : Exploring the details of a test level throughout the SPL lifecycle and illustrating the challenges associated with neglecting a particular test level would provide valuable insights for practitioners (Sect. 4.2 ). Two levels of tests are commonly executed throughout Domain Engineering: Unit testing and Integration testing. Although testing common core assets of an SPL is vital to detect faults as soon as possible, a few studies have considered the execution of tests in domain engineering. Therefore, it would be useful to conduct further investigations regarding how to execute a specific level of test in Domain Engineering and the consequences of not performing it. In Application Engineering, three levels of tests are usually executed: Unit testing, Integration testing, and System/acceptance testing. The two last levels have been investigated in most of the studies. It is worth mentioning that Unit testing has been investigated as a level of test in Application Engineering in a few studies published in recent years. In contrast, previous literature reviews have not reported this level of test in Application Engineering (e.g., Pérez et al. 2009 ). This indicates no consensus on the test levels executed during Domain Engineering and Application Engineering.

Another aspect that needs further exploration involves examining the influence of variabilities inherent in SPLs on different test levels. This requires understanding how the presence of variable features across products affects test activities, including planning, design, and execution at each testing level. Additionally, there is a need to investigate how test levels adapt to requirements and feature set changes throughout the SPL lifecycle. This requires exploring the challenges and opportunities associated with maintaining effective testing strategies in response to the dynamic nature of evolving product configurations.

Preserving the traceability between test assets and development artifacts : Preserving traceability between test assets and development artifacts in SPLs is particularly challenging due to the complex relationships between product variants and the shared assets (Sect. 4.5 ). Studies that target testing SPLs (very) rarely consider traceability explicitly. Examining the challenges associated with preserving traceability is crucial, especially when dealing with evolving product configurations within the SPL testing environment. While researchers have proposed certain methods, such as Reis et al. ( 2007 ) which preserved the traceability between requirements and test case scenarios using UML models and by refining use case scenarios into test case scenarios, Reuys et al. ( 2006 ) enabled traceability between artifacts, there remains a necessity to investigate more efficient approaches for modeling and representing traceability relationships, considering feature variability and configuration management. Furthermore, exploring the creation of automated tools and techniques for establishing and consistently updating traceability links in response to the evolving nature of SPLs presents an engaging area for future research.

To compare findings with previous SLRs, Table 12 presents a summary of the findings from both the current study and prior literature reviews (Pérez et al. 2009 ; Engström and Runeson 2011 ; Da Mota Silveira Neto et al. 2011 ; do Carmo Machado et al. 2014 ).

7 Related work

This research aims to provide researchers and practitioners with an overview of state-of-the-art testing practices applied to SPL and identify the gaps between required techniques and existing approaches. Accordingly, we conducted an SLR to analyze existing approaches to SPL testing. Therefore, SLRs and SMSs on SPL testing can be considered as works related to this research. To the best of our knowledge, four papers have systematically analyzed approaches focused on SPL testing (Pérez et al. 2009 ; Engström and Runeson 2011 ; Da Mota Silveira Neto et al. 2011 ; do Carmo Machado et al. 2014 ).

Pérez et al. ( 2009 ) conducted an SLR to identify experience reports and initiatives carried out in the SPL testing area. In this work, primary studies were classified into seven categories: Unit testing, Integration testing, functional testing, SPL Architecture testing, Embedded system testing, testing process and testing effort in SPL. Then, they presented a summary of each area. The similarity of this SLR to our work is testing levels investigated in both works; however, our work is broader in scope than this SLR since we investigated more aspects of SPL testing.

Engström and Runeson ( 2011 ) conducted an SMS by analyzing papers published up to 2008. The authors mapped studies into seven categories based on their research focus: Test organization and process, Test management, Testability, System and acceptance testing, Integration testing, Unit testing, and Test automation. They also identified challenges in SPL testing and needs for future research. This SMS has similarities with our work regarding specific SPL aspects investigated, including testing levels and test automation. However, the research questions designed by Engström and Runeson ( 2011 ) are more general, focusing on specifying challenges and topics investigated in SPL testing.

Da Mota Silveira Neto et al. ( 2011 ) conducted an SMS to investigate state-of-the-art testing practices by analyzing a set of 45 publications dated from 1993 to 2009. Primary studies are mapped into nine categories: Testing strategy, Static and dynamic analysis, Testing levels, Regression testing, Non-functional testing, Commonality and variability testing, Variant binding time, Effort reduction, and Test measurement. Some of the research questions designed by Da Mota Silveira Neto et al. ( 2011 ) are similar to the ones investigated in our work (e.g., testing SPLs while considering commonalities and variabilities). However, our work is broader in scope since we analyzed 110 papers published up to 2022. Furthermore, we only included empirically evaluated studies in our review.

do Carmo Machado et al. ( 2014 ) conducted an SLR by analyzing 49 studies published up to 2013; this SLR aimed to identify testing strategies that could achieve higher defect detection rates and reduced quality assurance effort. Identifying strategies to handle the selection of products to test has been investigated in both (do Carmo Machado et al. 2014 ) and our work. Furthermore, similar to our work, the initial set of primary studies in study (do Carmo Machado et al. 2014 ) has been identified by investigating previously conducted SLRs or SMSs, published up to the year 2009; also, the authors of this SLR only included empirically evaluated studies. However, our work investigates more aspects of SPL testing (e.g., preserving traceability between test assets and other artifacts) and analyzes more studies (110 papers).

Literature reviews also specifically focused on analyzing one aspect of SPL testing. As an example, Lopez-Herrejon et al. ( 2015 ) conducted an SMS to identify techniques that have been applied for combinatorial interaction testing of SPLs. However, our work is broader in scope since we did not limit the studies to a specific technique.

In general, the previous literature reviews and our work complement each other regarding the research questions addressed. Some aspects of SPL testing have not been considered in detail in previous reviews: techniques used for preserving traceability between test artifacts and other artifacts, techniques employed for identifying valid and invalid configurations, and different ways to control cost/effort of SPL testing were not covered in an extent that makes it possible to identify the current status of research and practice from the perspective of those aspects.

8 Conclusions and future work

The goal of SPLE is to improve the effectiveness and efficiency of software development by managing commonalities and variabilities among products. Testing is an essential part of SPLE to achieve the benefits of an SPL. It is focused on detecting potential faults in core assets created during Domain Engineering and products created during Application Engineering by reusing core assets. This paper presents the results of a systematic literature review of testing in SPLE. The SLR aimed to investigate specific aspects of SPL testing that were formulated as seven research questions, identify gaps, and address specific points of SPLE that still need to be fully addressed.

The analysis that we conducted based on 118 studies from 2003 to 2022 has uncovered a range of issues and considerations that researchers and practitioners can work on. It is shown that managing variability in SPL testing is vital but can complicate the testing process. Model-based methods show promise in generating test assets, but there is room for improvement in automating test case creation and ensuring comprehensive coverage. Non-functional testing aspects like performance, security, and usability require more attention and specific methodologies. Having the right tools is important, but most tool implementations are still in the proof-of-concept stage. Regression testing poses a complex challenge, and future research should concentrate on areas like regression test selection, prioritization, minimization, and architecture-based regression testing. Establishing benchmark datasets and standard evaluation criteria for SPL testing methods would simplify comparing and adopting various techniques.

Exploring test levels throughout the SPL lifecycle and illustrating the challenges of neglecting a particular test level would offer valuable insights. Additionally, studies focusing on testing SPLs need to address traceability explicitly. Maintaining traceability between test assets and development artifacts is especially difficult due to the intricate relationships between product variants and shared assets, which requires effective approaches. It is also worth mentioning that, throughout selecting studies for final analysis, we included only the studies empirically evaluated. By analyzing the evaluation conducted in the studies, we noticed that most of the studies were assessed by applying only one empirical method. Furthermore, most of the assessments undertaken do not refer to real-world practice. This indicates the need to evaluate SPL testing approaches not in academia but in industry.

Based on the findings of this SLR, further research in the SPL testing field can be expended on specific areas we identified throughout this research as the potential points for future research (e.g., SPL regression testing). Furthermore, empirical assessment of existing techniques for the investigated aspects (e.g., selection of products to test or creating reusable test assets) to compare those techniques would be helpful for both researchers and practitioners, mainly if those techniques are applied to real-world and large-scale scenarios. Furthermore, this research can be strengthened by examining studies published in the field of testing configurable systems. Such analysis can investigate how techniques from this broader domain might be applied to SPL testing to address existing deficiencies in this area.

Data availability

All data generated during this study are available in the “Zenodo” repository: https://zenodo.org/doi/10.5281/zenodo.10018266 .

Replication package available on https://zenodo.org/doi/ https://doi.org/10.5281/zenodo.10018266 .

SPLC stands for Software Product Line Conference.

Aduni Sulaiman R, Jawawi DN, Halim SA (2019) Derivation of test cases for model-based testing of software product line with hybrid heuristic approach. In: IRICT’19, pp 199–208. https://doi.org/10.1007/978-3-030-33582-3_19

Akbari Z, Khoshnevis S, Mohsenzadeh M (2017) A method for prioritizing integration testing in software product lines based on feature model. Int J Softw Eng Knowl Eng 27(04):575–600. https://doi.org/10.1142/S0218194017500218

Article Google Scholar

Akimoto H, Isogami Y, Kitamura T, Noda N, Kishi T (2019) A prioritization method for SPL pairwise testing based on user profiles. In: APSEC’19, pp 118–125. https://doi.org/10.1109/APSEC48747.2019.00025

Al-Dallal J, Sorenson PG (2008) Testing software assets of framework-based product families during application engineering stage. J Softw 3(5):11–25

Google Scholar

Al-Hajjaji M, Krieter S, Thüm T, Lochau M, Saake G (2016) IncLing: efficient product-line testing using incremental pairwise sampling. ACM SIGPLAN Not 52(3):144–155. https://doi.org/10.1145/3093335.2993253

Al-Hajjaji M, Krüger J, Schulze S, Leich T, Saake G (2017a) Efficient product-line testing using cluster-based product prioritization. In: AST’17, pp 16–22. https://doi.org/10.1109/AST.2017.7

Al-Hajjaji M, Lity S, Lachmann R, Thüm T, Schaefer I, Saake G (2017b) Delta-oriented product prioritization for similarity-based product-line testing. In: VACE’17, pp 34–40. https://doi.org/10.1109/VACE.2017.8

Al-Hajjaji M, Thüm T, Lochau M, Meinicke J, Saake G (2019) Effective product-line testing using similarity-based product prioritization. Softw Syst Model 18(1):499–521. https://doi.org/10.1007/s10270-016-0569-2

AL-Msie’deen RF, Seriai A, Huchard M, Urtado C, Vauttier S, Salman HE (2013) Feature location in a collection of software product variants using formal concept analysis. In: ICSR’13, pp 302–307. https://doi.org/10.1007/978-3-642-38977-1_22

Alves V, Niu N, Alves C, Valença G (2010) Requirements engineering for software product lines: a systematic literature review. Inf Softw Technol 52(8):806–820

Alves Pereira J, Acher M, Martin H, Jézéquel JM (2020) Sampling effect on performance prediction of configurable systems: A case study. In: ICPE’20, pp 277–288. https://doi.org/10.1145/3358960.3379137

Ammann P, Offutt J (2008) Introduction to software testing. Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511809163

Book Google Scholar

Ampatzoglou A, Bibi S, Avgeriou P, Verbeek M, Chatzigeorgiou A (2019) Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf Softw Technol 106:201–230

Aoyama Y, Kuroiwa T, Kushiro N (2021) Executable test case generation from specifications written in natural language and test execution environment. In: CCNC’21, pp 1–6. https://doi.org/10.1109/CCNC49032.2021.9369549

Apel S, Batory D, Kästner C, Saake G (2013) Feature-oriented software product lines: concepts and implementation. Springer, Berlin

Araújo IL, Santos IS, Filho JB, Andrade RM, Neto PS (2017) Generating test cases and procedures from use cases in dynamic software product lines. In: SAC’17, pp 1296–1301. https://doi.org/10.1145/3019612.3019790

Arrieta A, Sagardui G, Etxeberria L (2015) Test control algorithms for the validation of cyber-physical systems product lines. In: SPLC’15, pp 273–282. https://doi.org/10.1145/2791060.2791095

Arrieta A, Wang S, Sagardui G, Etxeberria L (2016) Search-based test case selection of cyber-physical system product lines for simulation-based validation. In: SPLC’16, pp 297–306. https://doi.org/10.1145/2934466.2946046

Arrieta A, Wang S, Sagardui G, Etxeberria L (2019) Search-based test case prioritization for simulation-based testing of cyber-physical system product lines. J Syst Softw 149:1–34. https://doi.org/10.1016/j.jss.2018.09.055

Baller H, Lity S, Lochau M, Schaefer I (2014) Multi-objective test suite optimization for incremental product family testing. In: ICST’14, pp 303–312. https://doi.org/10.1109/ICST.2014.43

Belli F, Tuglular T, Ufuktepe E (2021) Heterogeneous modeling and testing of software product lines. In: QRS-C’21, pp 1079–1088. https://doi.org/10.1109/QRS-C55045.2021.00162

Beohar H, Mousavi MR (2016) Input–output conformance testing for software product lines. J Log Algebr Methods Program 85(6):1131–1153. https://doi.org/10.1016/j.jlamp.2016.09.007

Article MathSciNet Google Scholar

Bharathi M, Sangeetha V (2018) Weighted rank ant colony metaheuristics optimization-based test suite reduction in combinatorial testing for improving software quality. In: ICICCS’18, pp 525–534. https://doi.org/10.1109/ICCONS.2018.8663102

Bucaioni A, Di Silvestro F, Singh I, Saadatmand M, Muccini H (2022) Model-based generation of test scripts across product variants: an experience report from the railway industry. J Softw Evol Process 34(11):e2498. https://doi.org/10.1002/smr.2498

Bürdek J, Lochau M, Bauregger S, Holzer A, Rhein AV, Apel S, Beyer D (2015) Facilitating reuse in multi-goal test-suite generation for software product lines. In: FASE’15, pp 84–99. https://doi.org/10.1007/978-3-662-46675-9_6

Cabral I, Cohen MB, Rothermel G (2010) Improving the testing and testability of software product lines. In: SPLC’10, pp 241–255. https://doi.org/10.1007/978-3-642-15579-6_17

Calvagna A, Gargantini A, Vavassori P (2013) Combinatorial testing for feature models using CitLab. In: ICSTW’13, pp 338–347. https://doi.org/10.1109/ICSTW.2013.45

Carlsson M, Gotlieb A, Marijan D (2016) Software product line test suite reduction with constraint optimization. In: ICSOFT’16, pp 68–87. https://doi.org/10.1007/978-3-319-62569-0_4

Catal C, Mishra D (2012) Test case prioritization: a systematic mapping study. Softw Qual J 21(3):445–478. https://doi.org/10.1007/s11219-012-9181-z

Chen L, Babar MA (2011) A systematic review of evaluation of variability management approaches in software product lines. Inf Softw Technol 53(4):344–362. https://doi.org/10.1016/j.infsof.2010.12.006

Clarke D, Helvensteijn M, Schaefer I (2010) Abstract delta modeling. ACM SIGPLAN Not 46(2):13–22. https://doi.org/10.1145/1942788.1868298

Clements P, Northrop L (2002) Software product lines: practices and patterns. Addison-Wesley, Boston

Cmyrev A, Reissing R (2014) Efficient and effective testing of automotive software product lines. Appl Sci Eng Prog 7(2):53–57. https://doi.org/10.14416/j.ijast.2014.05.001

Cohen MB, Dwyer MB, Shi J (2006) Coverage and adequacy in software product line testing. In: ROSATEA’06, pp 53–63. https://doi.org/10.1145/1147249.1147257

Cohen MB, Dwyer MB, Shi J (2007) Interaction testing of highly-configurable systems in the presence of constraints. In: ISSTA’07, pp 129–139. https://doi.org/10.1145/1273463.1273482

Cruzes DS, Dybä T (2011) Research synthesis in software engineering: a tertiary study. Inf Softw Technol 53(5):440–455. https://doi.org/10.1016/j.infsof.2011.01.004

Czarnecki K, Eisenecker UW (2000) Generative programming: methods, tools and applications. Addison-Wesley, New York

Da Mota Silveira Neto PA, do, Carmo Machado I, McGregor JD, De Almeida ES, de Lemos Meira SR (2011) A systematic mapping study of software product lines testing. Inf Softw Technol 53(5):407–423. https://doi.org/10.1016/j.infsof.2010.12.003

Denger C, Kolb R (2006) Testing and inspecting reusable product line components: First empirical results. In: ISESE’06, pp 184–193. https://doi.org/10.1145/1159733.1159762

Devroey X, Perrouin G, Legay A, Schobbens PY, Heymans P (2017) Dissimilar test case selection for behavioural software product line testing, In: SPLC’17, pp 1–9

do Carmo Machado I, McGregor JD, Cavalcanti YC, De Almeida ES (2014) On strategies for testing software product lines: a systematic literature review. Inf Softw Technol 56(10):1183–1199. https://doi.org/10.1016/j.infsof.2014.04.002

do Nascimento Ferreira T, Kuk JN, Pozo A, Vergilio SR (2016) Product selection based on upper confidence bound MOEA/D-DRA for testing software product lines. In: CEC’16, pp 4135–4142. https://doi.org/10.1109/CEC.2016.7744315

Dominka S, Mandl M, Dübner M, Ertl D (2018) Using combinatorial testing for distributed automotive features: Applying combinatorial testing for automated feature-interaction-testing. In: CCWC’18, pp 490–495. https://doi.org/10.1109/CCWC.2018.8301632

Drave I, Hillemacher S, Greifenberg T, Kriebel S, Kusmenko E, Markthaler M, Orth P, Salman KS, Richenhagen J, Rumpe B, Schulze C (2019) SMArDT modeling for automotive software testing. Softw Pract Exp 49(2):301–328. https://doi.org/10.1002/spe.2650

Dybå T, Dingsøyr T (2008) Empirical studies of agile software development: a systematic review. Inf Softw Technol 50(9–10):833–859. https://doi.org/10.1016/j.infsof.2008.01.006

Ebert R, Jolianis J, Kriebel S, Markthaler M, Pruenster B, Rumpe B, Salman KS (2019) Applying product line testing for the electric drive system. In: SPLC’19, pp 14–24. https://doi.org/10.1145/3336294.3336318

Engström E, Runeson P (2011) Software product line testing–A systematic mapping study. Inf Softw Technol 53(1):2–13. https://doi.org/10.1016/j.infsof.2010.05.011

Ensan F, Bagheri E, Gašević D (2012) Evolutionary search-based test generation for software product line feature models. In: CAiSE’12, pp 613–628. https://doi.org/10.1007/978-3-642-31095-9_40

Ergun B, Gebizli CŞ, Sözer H (2017) FORMAT: A tool for adapting test models based on feature models. In: COMPSAC’17, pp 66–71. https://doi.org/10.1109/COMPSAC.2017.134

Ferreira JM, Vergilio SR, Quináia MA (2013) A mutation approach to feature testing of software product lines. In: SEKE’13, pp 231–237

Ferreira JM, Vergilio SR, Quinaia MA (2017a) Software product line testing based on feature model mutation. Int J Softw Eng Knowl Eng 27(05):817–839. https://doi.org/10.1142/S0218194017500309

Ferreira TN, Vergilio SR, de Souza JT (2017b) Incorporating user preferences in search-based software engineering: a systematic mapping study. Inf Softw Technol 90:55–69. https://doi.org/10.1016/j.infsof.2017.05.003

Ferrer J, Chicano F, Alba E (2017) Hybrid algorithms based on integer programming for the search of prioritized test data in software product lines. In: EvoCOP’17, pp 3–19. https://doi.org/10.1007/978-3-319-55792-2_1

Ferrer J, Chicano F, Ortega-Toro JA (2021) CMSA algorithm for solving the prioritized pairwise test data generation problem in software product lines. J Heuristics 27(1):229–249. https://doi.org/10.1007/s10732-020-09462-w

Fischer S, Linsbauer L, Egyed A, Lopez-Herrejon RE (2018) Predicting higher order structural feature interactions in variable systems. In: ICSME’18, pp 252–263. https://doi.org/10.1109/ICSME.2018.00035

Fomel S (2002) Object management group: UML profile for schedulability, performance and time specification. OMG Doc 2(03):1–101

Fragal VH, Simao A, Endo AT, Mousavi MR (2017) Reducing the concretization effort in FSM-based testing of software product lines. In: ICSTW’17, pp 329–336. https://doi.org/10.1109/ICSTW.2017.61

Fragal VH, Simao A, Mousavi MR, Turker UC (2019) Extending HSI test generation method for software product lines. Comput J 62(1):109–129. https://doi.org/10.1093/comjnl/bxy046

Galindo JA, Alférez M, Acher M, Baudry B, Benavides D (2014) A variability-based testing approach for synthesizing video sequences. In: ISSTA’14, pp 293–303. https://doi.org/10.1145/2610384.2610411

Galindo JA, Turner H, Benavides D, White J (2016) Testing variability-intensive systems using automated analysis: an application to Android. Softw Qual J 24(2):365–405. https://doi.org/10.1007/s11219-014-9258-y

Gebizli CS, Sözer H (2016) Model-based software product line testing by coupling feature models with hierarchical markov chain usage models. In: QRS-C’16, pp 278–283. https://doi.org/10.1109/QRS-C.2016.42

Ghanam Y, Andreychuk D, Maurer F (2010) Reactive variability management in agile software development. In: 2010 Agile Conference, pp 27–34. https://doi.org/10.1109/AGILE.2010.6

Hajri I, Goknil A, Pastore F, Briand LC (2020) Automating system test case classification and prioritization for use case-driven testing in product lines. Empir Softw Eng 25(5):3711–3769. https://doi.org/10.1007/s10664-020-09853-4

Henard C, Papadakis M, Perrouin G, Klein J, Traon YL (2013) Multi-objective test generation for software product lines. In: SPLC’13, pp 62–71. https://doi.org/10.1145/2491627.2491635

Henard C, Papadakis M, Perrouin G, Klein J, Heymans P, Le Traon Y (2014a) Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans Softw Eng 40(7):650–670. https://doi.org/10.1109/TSE.2014.2327020

Henard C, Papadakis M, Traon YL (2014b) Mutation-based generation of software product line test configurations. In: SSBSE’14, pp 92–106. https://doi.org/10.1007/978-3-319-09940-8_7

Hentze M, Pett T, Sundermann C, Krieter S, Thüm T, Schaefer I (2022) Generic Solution-Space Sampling for Multi-domain Product Lines. In: GPCE’22, pp 135–147. https://doi.org/10.1145/3564719.3568695

Hervieu A, Baudry B, Gotlieb A (2011) PACOGEN: Automatic generation of pairwise test configurations from feature models. In: ISSRE’11, pp 120–129. https://doi.org/10.1109/ISSRE.2011.31

Hervieu A, Marijan D, Gotlieb A, Baudry B (2016) Practical minimization of pairwise-covering test configurations using constraint programming. Inf Softw Technol 71:129–146. https://doi.org/10.1016/j.infsof.2015.11.007

Hierons RM, Li M, Liu X, Parejo JA, Segura S, Yao X (2020) Many-objective test suite generation for software product lines. ACM Trans Softw Eng Methodol 29(1):1–46. https://doi.org/10.1145/3361146

Ibias A, Llana L, Núñez M (2022) Using ant colony optimisation to select features having associated costs. In: ICTSS’22, pp 106–122. https://doi.org/10.1007/978-3-031-04673-5_8

Jackson D (2012) Software abstractions: logic, language, and analysis. MIT Press

Jakubovski Filho HL, Ferreira TN, Vergilio SR (2018) Incorporating user preferences in a software product line testing hyper-heuristic approach. In: CEC’18, pp 1–8. https://doi.org/10.1109/CEC.2018.8477803

Jakubovski Filho HL, Ferreira TN, Vergilio SR (2019) Preference based multi-objective algorithms applied to the variability testing of software product lines. J Syst Softw 151:194–209. https://doi.org/10.1016/j.jss.2019.02.028

Jaring M, Krikhaar RL, Bosch J (2008) Modeling variability and testability interaction in software product line engineering. In: ICCBSS’08, pp 120–129. https://doi.org/10.1109/ICCBSS.2008.9

Johansen MF, Haugen Ø, Fleurey F (2011) Properties of realistic feature models make combinatorial testing of product lines feasible. In: MODELS’11, pp 638–652. https://doi.org/10.1007/978-3-642-24485-8_47

Jorgensen PC (2013) Software testing: a craftsman’s approach. Auerbach Publications

Jung P, Kang S, Lee J (2019) Automated code-based test selection for software product line regression testing. J Syst Softw 158:110419. https://doi.org/10.1016/j.jss.2019.110419

Jung P, Kang S, Lee J (2020) Efficient regression testing of software product lines by reducing redundant test executions. Appl Sci 10(23):8686. https://doi.org/10.3390/app10238686

Jung P, Kang S, Lee J (2022) Reducing redundant test executions in software product line testing—A case study. Electronics 11(7):1165. https://doi.org/10.3390/electronics11071165

Käkölä T, Dueñas JC (2006) Research issues in software product lines—Engineering and management. Springer, Heidelberg

Kang S, Baek H, Kim J, Lee J (2015) Systematic software product line test case derivation for test data reuse. In: COMPSAC’15, pp 433–440. https://doi.org/10.1109/COMPSAC.2015.174

Kim CH, Batory DS, Khurshid S (2011) Reducing combinatorics in testing product lines. In: AOSD’11, pp 57–68. https://doi.org/10.1145/1960275.1960284

Kim CH, Khurshid S, Batory D (2012) Shared execution for efficiently testing product lines. In: ISSRE’12, pp 221–230. https://doi.org/10.1109/ISSRE.2012.23

Kim CH, Marinov D, Khurshid S, Batory D, Souto S, Barros P, d’Amorim M (2013) SPLat: Lightweight dynamic analysis for reducing combinatorics in testing configurable systems. In: ESEC/FSE’13, pp 257–267. https://doi.org/10.1145/2491411.2491459

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report, Keele University and Durham University

Kitchenham B, Budgen D, Brereton P (2016) Evidence-based software engineering and systematic reviews. CRC Press

Krieter S, Thüm T, Schulze S, Saake G, Leich T (2020) YASA: Yet another sampling algorithm. In: VaMoS’20, pp 1–10. https://doi.org/10.1145/3377024.3377042

Kuhn DR, Wallace DR, Gallo AM (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421. https://doi.org/10.1109/TSE.2004.24

Kumar S (2016) Test case prioritization techniques for software product line: A survey. In: ICCCA, pp 884–889. https://doi.org/10.1109/CCAA.2016.7813841

Lachmann R, Lity S, Lischke S, Beddig S, Schulze S, Schaefer I (2015) Delta-oriented test case prioritization for integration testing of software product lines. In: SPLC’15, pp 81–90. https://doi.org/10.1145/2791060.2791073

Lachmann R, Lity S, Al-Hajjaji M, Fürchtegott F, Schaefer I (2016) Fine-grained test case prioritization for integration testing of delta-oriented software product lines. In: FOSD’16, pp 1–10. https://doi.org/10.1145/3001867.3001868

Lachmann R, Beddig S, Lity S, Schulze S, Schaefer I (2017) Risk-based integration testing of software product lines. In: VaMoS’17, pp 52–59. https://doi.org/10.1145/3023956

Lackner H, Thomas M, Wartenberg F, Weißleder S (2014) Model-based test design of product lines: Raising test design to the product line level. In: ICST’14, pp 51–60. https://doi.org/10.1109/ICST.2014.16

Lamancha BP, Polo M, Piattini M (2015) PROW: a pairwise algorithm with constraints, order and weight. J Syst Softw 99:1–19. https://doi.org/10.1016/j.jss.2014.08.005

Lee J, Hwang S (2019) Combinatorial test design using design-time decisions for variability. Int J Softw Eng Knowl Eng 29(08):1141–1158. https://doi.org/10.1142/S0218194019400138

Li Z, Harman M, Hierons RM (2007) Search algorithms for regression test case prioritization. IEEE Trans Softw Eng 33(4):225–237. https://doi.org/10.1109/TSE.2007.38

Li X, Wong WE, Gao R, Hu L, Hosono S (2018) Genetic algorithm-based test generation for software product line with the integration of fault localization techniques. Empir Softw Eng 23(1):1–51. https://doi.org/10.1007/s10664-016-9494-9

Lima JA, Mendonça WD, Vergilio SR, Assunção WK (2020) Learning-based prioritization of test cases in continuous integration of highly-configurable software. In: SPLC’20, pp 1–11. https://doi.org/10.1145/3382025.3414967

Lity S, Nieke M, Thüm T, Schaefer I (2019) Retest test selection for product-line regression testing of variants and versions of variants. J Syst Softw 147:46–63. https://doi.org/10.1016/j.jss.2018.09.090

Lochau M, Oster S, Goltz U, Schürr A (2012a) Model-based pairwise testing for feature interaction coverage in software product line engineering. Softw Qual J 20(3):567–604. https://doi.org/10.1007/s11219-011-9165-4

Lochau M, Schaefer I, Kamischke J, Lity S (2012b) Incremental model-based testing of delta-oriented software product lines. In: TAP’12, pp 67–82. https://doi.org/10.1007/978-3-642-30473-6_7

Lochau M, Lity S, Lachmann R, Schaefer I, Goltz U (2014) Delta-oriented model-based integration testing of large-scale systems. J Syst Softw 91:63–84. https://doi.org/10.1016/j.jss.2013.11.1096

Lopez-Herrejon RE, Javier Ferrer J, Chicano F, Haslinger EN, Egyed A, Alba E (2014) A parallel evolutionary algorithm for prioritized pairwise testing of software product lines. In: GECCO’14, pp 1255–1262. https://doi.org/10.1145/2576768.2598305

Lopez-Herrejon RE, Fischer S, Ramler R, Egyed A (2015) A first systematic mapping study on combinatorial interaction testing for software product lines. In: ICSTW’15, pp 1–10. https://doi.org/10.1109/ICSTW.2015.7107435

Luo L (2001) Software testing techniques. Institute for Software Research International Carnegie Mellon University. Pittsburgh PA 15232(19):1–19

Luo G, Petrenko A, Bochmann GV (1995) Selecting test sequences for partially-specified nondeterministic finite state machines. In: IFIP WG, pp 95–110. https://doi.org/10.1007/978-0-387-34883-4_6

Luthmann L, Gerecht T, Stephan A, Bürdek J, Lochau M (2019a) Minimum/maximum delay testing of product lines with unbounded parametric real-time constraints. J Syst Softw 149:535–553. https://doi.org/10.1016/j.jss.2018.12.028

Luthmann L, Gerecht T, Lochau M (2019b) Sampling strategies for product lines with unbounded parametric real-time constraints. Int J Softw Tools Technol Transf 21(6):613–633. https://doi.org/10.1007/s10009-019-00532-4

Marijan D, Gotlieb A, Sen S, Hervieu A (2013) Practical pairwise testing for software product lines. In: SPLC’13, pp 227–235. https://doi.org/10.1145/2491627.2491646

Marijan D, Liaaen M, Gotlieb A, Sen S, Ieva C (2017) Titan: Test suite optimization for highly configurable software. In: ICST’17, pp 524–531. https://doi.org/10.1109/ICST.2017.60

Markiegi U, Arrieta A, Sagardui G, Etxeberria L (2017) Search-based product line fault detection allocating test cases iteratively. In: SPLC’17, pp 123–132. https://doi.org/10.1145/3106195.3106210

Markiegi U, Arrieta A, Etxeberria L, Sagardui G (2019) Test case selection using structural coverage in software product lines for time-budget constrained scenarios. In: SAC’19, pp 2362–2371. https://doi.org/10.1145/3297280.3297512

Matnei Filho RA, Vergilio SR (2016) A multi-objective test data generation approach for mutation testing of feature models. J Softw Eng Res Dev 4(1):1–29. https://doi.org/10.1186/s40411-016-0030-9

Mayring P (2014) Qualitative Content Analysis: Theoretical Foundation, Basic Procedures and Software Solution. Klagenfurt. Available at Social Science Open Access Repository (SSOAR) https://nbn-resolving.de/urn:nbn:de:0168-ssoar-395173 (accessed 04 June 2024)

McGregor JD (2001) Testing a software product line. Technical Report, Carnegie Mellon University

Mendes E, Wohlin C, Felizardo K, Kalinowski M (2020) When to update systematic literature reviews in software engineering. J Syst Softw 167:110607. https://doi.org/10.1016/j.jss.2020.110607

Mishra S (2006) Specification based software product line testing: A case study. In: CS&P’06, pp 243–254

Nebut C, Pickin S, Le Traon Y, Jézéquel JM (2003) Automated requirements-based generation of test cases for product families. In: ASE’03, pp 263–266. https://doi.org/10.1109/ASE.2003.1240317

Nebut C, Traon YL, Jézéquel JM (2006) System testing of product lines: from requirements to test cases. In: Käköla T, Duenas JC (eds) Software Product lines. Springer, Berlin, Heidelberg, pp 447–477. https://doi.org/10.1007/978-3-540-33253-4_12

Chapter Google Scholar

Neto PA, do Carmo Machado I, Cavalcanti YC, De Almeida ES, Garcia VC, de Lemos Meira SR (2010) A regression testing approach for software product lines architectures. In: SBCARS’10, pp 41–50. https://doi.org/10.1109/SBCARS.2010.14

Nguyen QL (2009) Non-functional requirements analysis modeling for software product lines. In: ICSE’09, pp 56–61. https://doi.org/10.1109/MISE.2009.5069898

Northrop L, Clements P, Bachmann F, Bergey J, Chastek G, Cohen S, Donohoe P, Jones L, Krut R, Little R (2007) A framework for software product line practice, version 5.0. Technical report, Carnegie Mellon University

Olimpiew EM, Gomaa H (2009) Reusable model-based testing. In: ICSR’09, pp 76–85. https://doi.org/10.1007/978-3-642-04211-9_8

Oster S, Markert F, Ritter P (2010) Automated incremental pairwise testing of software product lines. In: SPLC’10, pp 196–210. https://doi.org/10.1007/978-3-642-15579-6_14

Parejo JA, Sánchez AB, Segura S, Ruiz-Cortés A, Lopez-Herrejon RE, Egyed A (2016) Multi-objective test case prioritization in highly configurable systems: a case study. J Syst Softw 122:287–310. https://doi.org/10.1016/j.jss.2016.09.045

Parra C, Giral L, Infante A, Cortés C (2012) Extractive SPL adoption using multi-level variability modeling. In: SPLC’12, pp 99–106. https://doi.org/10.1145/2364412.2364429

Patel S, Gupta P, Shah V (2013) Combinatorial interaction testing with multi-perspective feature models. In: ICSTW’13, pp 321–330. https://doi.org/10.1109/ICSTW.2013.43

Pérez B, Polo M, Piatini M (2009) Software product line testing-A systematic review. In: ICSOFT’09, pp 1–8

Perrouin G, Sen S, Klein J, Baudry B, Le Traon Y (2010) Automated and scalable t-wise test case generation strategies for software product lines. In: ICST’10, pp 459–468. https://doi.org/10.1109/ICST.2010.43

Petry KL, OliveiraJr E, Zorzo AF (2020) Model-based testing of software product lines: mapping study and research roadmap. J Syst Softw 167:110608. https://doi.org/10.1016/j.jss.2020.110608

Pett T, Eichhorn D, Schaefer I (2020) Risk-based compatibility analysis in automotive systems engineering. In: MODELS’20, pp 1–10. https://doi.org/10.1145/3417990.3421263

Pohl K, Metzger A (2006) Software product line testing. Commun ACM 49(12):78–81. https://doi.org/10.1145/1183236.1183271

Pohl K, Böckle G, Van Der Linden F (2005) Software product line engineering: foundations, principles, and techniques. Springer Berlin, Heidelberg. https://doi.org/10.1007/3-540-28901-1

Qian Y, Zhang C, Wang F (2018) Selecting products for high-strength t-wise testing of software product line by multi-objective method. In: PIC’18, pp 370–378. https://doi.org/10.1109/PIC.2018.8706270

Reis S, Metzger A, Pohl K (2006) A reuse technique for performance testing of software product lines. In: SPLiT’06, pp 5–10

Reis S, Metzger A, Pohl K (2007) Integration testing in software product line engineering: a model-based technique. In: FASE’07, pp 321–335. https://doi.org/10.1007/978-3-540-71289-3_25

Reuling D, Bürdek J, Rotärmel S, Lochau M, Kelter U (2015) Fault-based product-line testing: Effective sample generation based on feature-diagram mutation. In: SPLC’15, pp 131–140. https://doi.org/10.1145/2791060.2791074

Reuys A, Kamsties E, Pohl K, Reis S (2005) Model-based system testing of software product families. In: CAiSE’05, pp 519–534. https://doi.org/10.1007/11431855_36

Reuys A, Reis S, Kamsties E, Pohl K (2006) The scented method for testing software product lines. In: SPLC’06, pp 479–520. https://doi.org/10.1007/978-3-540-33253-4_13

Rocha L, Machado I, Almeida E, Kästner C, Nadi S (2020) A semi-automated iterative process for detecting feature interactions. In: SBES’20, pp 778–787. https://doi.org/10.1145/3422392.3422418

Roggenbach M (2006) CSP-CASL—A new integration of process algebra and algebraic specification. Theor Comput Sci 354(1):42–71. https://doi.org/10.1016/j.tcs.2005.11.007

Rubin J, Chechik M (2013) A survey of feature location techniques. In: Reinhartz-Berger I, Sturm A, Clark T, Cohen S, Bettin J (eds) Domain Engineering. Springer, Berlin, Heidelberg, pp 29–58. https://doi.org/10.1007/978-3-642-36654-3_2

Safdar SA, Yue T, Ali S (2021) Recommending faulty configurations for interacting systems under test using multi-objective search. ACM Trans Softw Eng Methodol 30(4):1–36. https://doi.org/10.1145/3464939

Saini A, Rajkumar, Kumar S (2022) Software Product Line Testing—A Proposal of Distance-Based Approach. In: AISE’20, pp 187–198. https://doi.org/10.1007/978-981-16-8542-2_15

Sánchez AB, Segura S, Ruiz-Cortés A (2014) A comparison of test case prioritization criteria for software product lines. In: ICST’14, pp 41–50. https://doi.org/10.1109/ICST.2014.15

Schaefer I, Bettini L, Bono V, Damiani F, Tanzarella N (2010) Delta-oriented programming of software product lines. In: SPLC’10, pp 77–91. https://doi.org/10.1007/978-3-642-15579-6_6

Scheidemann KD (2006) Optimizing the selection of representative configurations in verification of evolving product lines of distributed embedded systems. In: SPLC’06, pp 75–84. https://doi.org/10.1109/SPLINE.2006.1691579

Shi J, Cohen MB, Dwyer MB (2012) Integration testing of software product lines using compositional symbolic execution. In: FASE’12, pp 270–284. https://doi.org/10.1007/978-3-642-28872-2_19

Sjoberg DI, Dyba T, Jorgensen M (2007) The future of empirical methods in software engineering research. In: FOSE’07, pp 358–378. https://doi.org/10.1109/FOSE.2007.30

Soe NT, Wild N, Tanachutiwat S, Lichter H (2022) Design and implementation of a test automation framework for configurable devices. In: APIT’22, pp 200–207. https://doi.org/10.1145/3512353.3512383

Souto S, d’Amorim M (2018) Time-space efficient regression testing for configurable systems. J Syst Softw 137:733–746. https://doi.org/10.1016/j.jss.2017.08.010

Souto S, d’Amorim M, Gheyi R (2017) Balancing soundness and efficiency for practical testing of configurable systems. In: ICSE’17, pp 632–642. https://doi.org/10.1109/ICSE.2017.64

Steffens M, Oster S, Lochau M, Fogdal T (2012) Industrial evaluation of pairwise SPL testing with MoSo-PoLiTe. In: VaMoS’12, pp 55–62. https://doi.org/10.1145/2110147.2110154

Stricker V, Metzger A, Pohl K (2010) Avoiding redundant testing in application engineering. In: SPLC’10, pp 226–240. https://doi.org/10.1007/978-3-642-15579-6_16

Strickler A, Lima JA, Vergilio SR, Pozo AT (2016) Deriving products for variability test of feature models with a hyper-heuristic approach. Appl Soft Comput 49:1232–1242. https://doi.org/10.1016/j.asoc.2016.07.059

Tevanlinna A, Taina J, Kauppinen R (2004) Product family testing: a survey. ACM SIGSOFT Softw Eng Notes 29(2):12–12. https://doi.org/10.1145/979743.979766

Tuglular T, Coşkun DE (2021) Behavior-driven development of software product lines. In: DSA’21, pp 230–239. https://doi.org/10.1109/DSA52907.2021.00035

Tuglular T, Beyazıt M, Öztürk D (2019) Featured event sequence graphs for model-based incremental testing of software product lines. In: COMPSAC’19, pp 197–202. https://doi.org/10.1109/COMPSAC.2019.00035

Uzuncaova E, Khurshid S, Batory D (2010) Incremental test generation for software product lines. IEEE Trans Softw Eng 36(3):309–322. https://doi.org/10.1109/TSE.2010.30

Varshosaz M, Al-Hajjaji M, Thüm T, Runge T, Mousavi MR, Schaefer I (2018) A classification of product sampling for software product lines. In: SPLC’18, pp 1–13. https://doi.org/10.1145/3233027.3233035

Vazirani VV (2001) Approximation algorithms. Springer, Berlin. https://doi.org/10.1007/978-3-662-04565-7

Vidács L, Horváth F, Mihalicza J, Vancsics B, Beszédes Á (2015) Supporting software product line testing by optimizing code configuration coverage. In: ICSTW’15, pp 1–7. https://doi.org/10.1109/ICSTW.2015.7107478

Vidal Silva C, Galindo Duarte JÁ, Benavides Cuevas DF (2020) Functional testing of conflict detection and diagnosis tools in feature model configuration: a test suite design. In: ConfWS’20, pp 17–24

Wang S, Buchmann D, Ali S, Gotlieb A, Pradhan D, Liaaen M (2014) Multi-objective test prioritization in software product line testing: an industrial case study. In: SPLC’14, pp 32–41. https://doi.org/10.1145/2648511.2648515

Wang S, Ali S, Gotlieb A (2015) Cost-effective test suite minimization in product lines using search techniques. J Syst Softw 103:370–391. https://doi.org/10.1016/j.jss.2014.08.024

Wang S, Ali S, Gotlieb A, Liaaen M (2017) Automated product line test case selection: industrial case study and controlled experiment. Softw Syst Model 16(2):417–441. https://doi.org/10.1007/s10270-015-0462-4

Webster J, Watson RT (2002) Analyzing the past to prepare for the future: writing a literature review. MIS Q 26(2):xiii–xxiii

Weiss DM (2008) The product line hall of fame. In: SPLC’08, pp 39. https://doi.org/10.1109/SPLC.2008.56

Wohlin C, Höst M, Henningsson K (2003) Empirical research methods in software engineering. Empirical methods and studies in Software Engineering-experiences. Springer, Berlin, Heidelberg, pp 7–23. https://doi.org/10.1007/978-3-540-45143-3_2

Xiang Y, Huang H, Zhou Y, Li S, Luo C, Lin Q, Yang X (2022) Search-based diverse sampling from real-world software product lines. In: ICSE’22, pp 1945–1957. https://doi.org/10.1145/3510003.3510053

Xu Z, Cohen MB, Motycka W, Rothermel G (2013) Continuous test suite augmentation in software product lines. In: SPLC’13, pp 52–61. https://doi.org/10.1145/2491627.2491650

Yan L, Hu W, Han L (2019) Optimize SPL test cases with adaptive simulated annealing genetic algorithm. In: ACM TURC’19, pp 1–7. https://doi.org/10.1145/3321408.3326676

Yu L, Duan F, Lei Y, Kacker RN, Kuhn DR (2014) Combinatorial test generation for software product lines using minimum invalid tuples. In: HASE’14, pp 65–72. https://doi.org/10.1109/HASE.2014.18

Zhang L, Tian JH, Jiang J, Liu YJ, Pu MY, Yue T (2018) Empirical research in software engineering-A literature survey. JCST 33(5):876–899. https://doi.org/10.1007/s11390-018-1864-x

Download references

This work was funded by the Ministerium für Wissenschaft, Forschung und Kunst Baden-Württemberg in the Innovation Campus Mobility of the Future, projects SWUpCar and TESSOF.

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Institute of Software Engineering, University of Stuttgart, Stuttgart, Germany

Halimeh Agh, Aidin Azamnouri & Stefan Wagner

TUM School of Communication, Information and Technology, Technical University of Munich, Heilbronn, Germany

Stefan Wagner

You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Halimeh Agh , Aidin Azamnouri or Stefan Wagner .

Ethics declarations

Conflict of interest.

The authors declare that they have no conflict of interest.

Additional information

Communicated by Sven Apel.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Table 17 shows the results of the evaluation based on the quality assessment criteria, described in Table 14 in Appendix B. Regarding the issue Reporting (QA1-QA3 in Table 14 ), most of the studies performed well; all the studies are based on research and almost 82% of them have a clear statement of the aims of the research. However, the description of the context is bad in some of the studies (∼ 30%); this compromises the validity of these studies since, without enough information about the subjects of the study, it is usually difficult to specify whether the selected case is suitable to evaluate different aspects of the proposed approach.

In terms of rigor (QA4-QA7), the studies performed, on average, fairly well. In 77 studies (∼ 62%), the researchers have justified the research design so that it can address the aims of the research. In 71 studies (∼ 60%), the proposed approach has been compared with a base approach; the researcher(s) has tried to justify that the selected controls are representative of a defined population. The way data collected is satisfactory in 85 studies (∼ 72%) since the researchers have clearly defined the measure(s) selected and justified their selection. Furthermore, the data has been analyzed rigorously in 80 studies (68%) by providing sufficient data to support the findings. Although these findings are promising, 32% of the studies, overall, fail in rigor; this compromises the validity and usefulness of these studies since failing in rigor, as a key issue in Evidence-Based Software Engineering, indicates that the empirical methods have been applied in an informal way.

Regarding the issue Credibility, 95% of the studies provide a clear statement of the findings (QA9) by discussing the findings in relation to the research questions and also presenting the limitations of the study. However, most studies perform poorly in establishing relationships between the researcher(s) and participants and the data collected to address the research issue (QA8); this quality attribute is considered in only 12 studies (∼ 10%). This can threaten the quality of the research due to not considering potential bias and influence of the researcher(s) during the formulation of research questions, data collection, and analysis and selection of data for presentation.

In terms of Relevance, 114 studies (∼ 97%) explicitly deal with SPL testing and discuss the contributions the study makes to existing knowledge, identify new areas in which research is necessary, and discuss the ways in which the research can be used (QA10). This result is in line with the nature of the research goals, described as inclusion and exclusion criteria in Sect. 3.2. However, only 18 studies (∼ 15%) present practitioner-based guidelines (QA11). This indicates that the SPL testing field needs more practical guidance to strengthen the adoption of industry.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Agh, H., Azamnouri, A. & Wagner, S. Software product line testing: a systematic literature review. Empir Software Eng 29 , 146 (2024). https://doi.org/10.1007/s10664-024-10516-x

Download citation

Accepted : 19 June 2024

Published : 02 September 2024

DOI : https://doi.org/10.1007/s10664-024-10516-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Software product lines
Software testing
Software quality
Systematic literature review
Find a journal
Publish with us
Track your research

Systematic Review
Open access
Published: 30 August 2024

A scoping review of stroke services within the Philippines

Angela Logan 1 , 2 ,
Lorraine Faeldon 3 ,
Bridie Kent 1 , 4 ,
Aira Ong 1 &
Jonathan Marsden 1

BMC Health Services Research volume 24 , Article number: 1006 ( 2024 ) Cite this article

2 Altmetric

Metrics details

Stroke is a leading cause of mortality and disability. In higher-income countries, mortality and disability have been reduced with advances in stroke care and early access to rehabilitation services. However, access to such services and the subsequent impact on stroke outcomes in the Philippines, which is a lower- and middle-income countries (LMIC), is unclear. Understanding gaps in service delivery and underpinning research from acute to chronic stages post-stroke will allow future targeting of resources.

This scoping review aimed to map available literature on stroke services in the Philippines, based on Arksey and O’Malley’s five-stage-process.

Summary of review

A targeted strategy was used to search relevant databases (Focused: MEDLINE (ovid), EMBASE (ovid), Cumulative Index to Nursing and Allied Health Literature (CINAHL), PsycINFO (ebsco); broad-based: Scopus; review-based: Cochrane Library, International Prospective Register of Systematic Reviews (PROSPERO), JBI (formerly Joanna Briggs Institute) as well as grey literature (Open Grey, Google scholar). The searches were conducted between 12/2022-01/2023 and repeated 12/2023. Literature describing adults with stroke in the Philippines and stroke services that aimed to maximize well-being, participation and function were searched. Studies were selected if they included one or more of: (a) patient numbers and stroke characteristics (b) staff numbers, qualifications and role (c) service resources (e.g., access to a rehabilitation unit) (d) cost of services and methods of payment) (e) content of stroke care (f) duration of stroke care/rehabilitation and interventions undertaken (g) outcome measures used in clinical practice.

A total of 70 papers were included. Articles were assessed, data extracted and classified according to structure, process, or outcome related information. Advances in stroke services, including stroke ready hospitals providing early access to acute care such as thrombectomy and thrombolysis and early referral to rehabilitation coupled with rehabilitation guidelines have been developed. Gaps exist in stroke services structure (e.g., low number of neurologists and neuroimaging, lack of stroke protocols and pathways, inequity of stroke care across urban and rural locations), processes (e.g., delayed arrival to hospital, lack of stroke training among health workers, low awareness of stroke among public and non-stroke care workers, inequitable access to rehabilitation both hospital and community) and outcomes (e.g., low government insurance coverage resulting in high out-of-pocket expenses, limited data on caregiver burden, absence of unified national stroke registry to determine prevalence, incidence and burden of stroke). Potential solutions such as increasing stroke knowledge and awareness, use of mobile stroke units, TeleMedicine, TeleRehab, improving access to rehabilitation, upgrading PhilHealth and a unified national long-term stroke registry representing the real situation across urban and rural were identified.

This scoping review describes the existing evidence-base relating to structure, processes and outcomes of stroke services for adults within the Philippines. Developments in stroke services have been identified however, a wide gap exists between the availability of stroke services and the high burden of stroke in the Philippines. Strategies are critical to address the identified gaps as a precursor to improving stroke outcomes and reducing burden. Potential solutions identified within the review will require healthcare government and policymakers to focus on stroke awareness programs, primary and secondary stroke prevention, establishing and monitoring of stroke protocols and pathways, sustainable national stroke registry, and improve access to and availability of rehabilitation both hospital and community.

What is already known?

Stroke services in the Philippines are inequitable, for example, urban versus rural due to the geography of the Philippines, location of acute stroke ready hospitals and stroke rehabilitation units, limited transport options, and low government healthcare insurance coverage resulting in high out-of-pocket costs for stroke survivors and their families.

What are the new findings?

The Philippines have a higher incidence of stroke in younger adults than other LMICs, which impacts the available workforce and the country’s economy. There is a lack of data on community stroke rehabilitation provision, the content and intensity of stroke rehabilitation being delivered and the role and knowledge/skills of those delivering stroke rehabilitation, unmet needs of stroke survivors and caregiver burden and strain,

What do the new findings imply?

A wide gap exists between the availability of stroke services and the high burden of stroke. The impact of this is unclear due to the lack of a compulsory national stroke registry as well as published data on community or home-based stroke services that are not captured/published.

What does this review offer?

This review provides a broad overview of existing evidence-base of stroke services in the Philippines. It provides a catalyst for a) healthcare government to address stroke inequities and burden; b) development of future evidence-based interventions such as community-based rehabilitation; c) task-shifting e.g., training non-neurologists, barangay workers and caregivers; d) use of digital technologies and innovations e.g., stroke TeleRehab, TeleMedicine, mobile stroke units.

Peer Review reports

Introduction

In the Philippines, stroke is the second leading cause of death, with a prevalence of 0·9% equating to 87,402 deaths per annum [ 1 , 2 ]. Approximately 500,000 Filipinos will be affected by stroke, with an estimated US$350 million to $1·2 billion needed to meet the cost of medical care [ 1 ]. As healthcare is largely private, the cost is borne out-of-pocket by patients and their families. This provides a major obstacle for the lower socio-demographic groups in the country.

Research on implementation of locally and regionally adapted stroke-services and cost-effective secondary prevention programs in the Philippines have been cited as priorities [ 3 , 4 ]. Prior to developing, implementing, and evaluating future context-specific acute stroke management services and community-based models of rehabilitation, it was important to map out the available literature on stroke services and characteristics of stroke in the Philippines.

The scoping review followed a predefined protocol, established methodology [ 5 ] and is reported according to the Preferred Reporting Items for Systematic Review and Meta-Analyses Extension for Scoping Reviews Guidelines (PRISMA-ScR) [ 6 , 7 ]. Healthcare quality will be described according to the following three aspects: structures, processes, and outcomes following the Donabedian model [ 8 , 9 ].The review is based on Arksey and O'Malley’s five stages framework [ 5 ].

Stage 1: The research question:

What stroke services are available for adults within the Philippines? The objective was to systematically scope the literature to describe the availability, structure, processes, and outcome of stroke services for adults within the Philippines.

Stage 2: Identifying relevant studies:

The following databases were searched. Focused: MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature (CINAHL), PsycINFO; broad-based: Scopus; review-based: Cochrane Library, Prospero, JBI (formerly Joanna Briggs Institute); Grey literature: Herdin, North Grey, Grey matters, MedRxiv, NIHR health technology assessment, Department of Health Philippines, The Kings Fund, Ethos, Carrot2. Additionally, reference lists of full text included studies were searched.

The targeted search strategy, developed in consultation with an information scientist, was adapted for each database (see supplemental data). Search terms were peer reviewed using the PRESS (Peer Review of Electronic Search Strategies) checklist [ 10 ].

The key search concepts from the Population, Concept and Context (PCC) framework were ≥ 18 years with a stroke living in the Philippines ( population ), stroke services aiming to maximize well-being, participation and function following a stroke ( concept ) and stroke services from acute to chronic including those involving healthcare professionals, non-healthcare related personnel or family or friends ( context ). Search tools such as medical subject headings (MESH) and truncation to narrow or expand searches were used. Single and combined search terms were included (see supplemental data). The search was initially conducted over two weeks in December 2022 and re-run in December 2023.

Studies were selected if they described stroke care in the Philippines in terms of one or more of the following: (a) patient numbers and stroke characteristics (b) staff numbers, qualifications and role (c) service resources (e.g., number of beds/access to a rehabilitation unit, equipment used) (d) cost of services and methods of payment (UHC, Insurance, private) (e) content of stroke care (f) duration of stroke care (hours of personnel contact e.g., Therapy hours per day); interventions undertaken (g) outcome measures used in clinical practice.

Additional criteria:

Context: all environments (home, hospital, outpatients, clinic, academic institute).

Date limits: published between 2002 onwards. This is based on the Philippines Community Rehabilitation Guidelines published in 2009 that would suggest that papers earlier than 2002 may not reflect current practice [ 11 ].

Qualitative and quantitative studies including grey literature.

Language: reported in English or Filipino only.

Publication status: no limit because the level of rigor was not assessed.

Type of study: no limit which included conference abstracts, as the level of rigor was not assessed.

Studies were excluded if they were in non-stroke populations or the full text article could not be obtained. Conference abstracts were excluded if there were insufficient data about methods and results.

Searches of databases were performed by one researcher (JM) and searches of grey literature were performed by one researcher (AO). All retrieved articles were uploaded into Endnote X9 software™, and duplicates identified and removed before transferring them to Rayyan [ 12 ] for screening.

Stage 3: study selection

The title and abstract were selected using eligibility criteria. Two pairs of researchers independently screened abstracts and titles;(Databases: JM and AL and grey literature by AO and LF). Where a discrepancy existed for title and abstract screening, the study was automatically included for full text review and discussed among reviewers.

Two reviewers (JM and AL) undertook full-text screening of the selected studies. Discrepancies were resolved through consensus discussions without the need for a third reviewer. There were no discrepancies that required a third reviewer. Reason for exclusion were documented according to pre-determined eligibility criteria. References of included full text articles were screened by each reviewer independently and identified articles were subjected to the same screening process as per the PRISMA-ScR checklist (Fig. 1 ).

PRISMA-ScR flow diagram

Stage 4: Charting the data

Two reviewers independently extracted the data using a piloted customized and standardized data extraction form including (1) Structure: financial (e.g., costs, insurance, government funding), resources (structure and number of stroke facilities, staff (number, profession/specialism, qualifications etc.), stroke characteristics (2) Process: duration of care, content of stroke care within acute, secondary care, community, outcome measures used; (3) Outcome: survival, function, patient satisfaction, cost (admission and interventions), and (4) year of publication, geographical location (including if Philippines only or multiple international locations) and type of evidence (e.g., policy, review, observational, experimental, clinical guidelines). Critical appraisal of included studies was not undertaken because the purpose of the review was to map available evidence on stroke services available within the Philippines.

Stage 5: Collating, summarising and reporting the results

The search identified 351 records from databases and registers. A total of 70 records are included and reasons for non-inclusion are summarized in Fig. 1 .

Study descriptors

The characteristics of included studies are shown in Supplementary Material Table 1. Of the 70 included studies, 36 were observational with most being based on a retrospective review of case notes ( n = 31), two were audits, eight were surveys or questionnaires, four were consensus opinion and/or guideline development, three were randomized controlled trial (RCT) or feasibility RCT, 1 was a systematic review, two were policy and guidelines, 11 were narrative reviews or opinion pieces, two were case series or reports and one was an experimental study.

Of the 70 studies, 32 (45.7%) were based in a single tertiary hospital site. There were only three papers based in the community (4.3%). Papers that were opinion pieces or reviews were classified as having a national focus. Of the 22 papers classified as having a national focus, 10 (45.5%) were narrative reviews/ opinion pieces (Table 1 ).

The primary focus of the research studies (excluding the 11 narrative reviews and 2 policy documents) were classified as describing structure ( n = 8, 14%); process ( n = 21,36.8%) or outcomes ( n = 29, 49.2%). The structure of acute care was described in seven studies out of eight studies ( n = 7/8 87.5%) whilst neurosurgery structures were described in one out of eight studies (12.5%). Acute care processes were described in 11 out of 21 studies ( n = 11/21 52.3%) whilst rehabilitation processes were described in six out of 21 studies (28.6%), with three out of 21 studies primarily describing outcome measurement (14.3%). The primary focus of the outcomes were stroke characteristics (25 out of 28 papers, 89.2%) in terms of number of stroke (prevalence), mortality or severity of stroke. Measures of stroke quality of life were not reported. Healthcare professional knowledge was described in two studies ( n = 2/28 7.1%) whilst risk factors for stroke were described in one study ( n = 1/28, 3.6%). Carer burden was described in one study ( n = 1/28, 3.6%).

A summary of the findings is presented in Table 2 .

This scoping review describes the available literature on stroke services within the Philippines across the lifespan of an adult (> 18 years) with a stroke. The review has identified gaps in information about structures, processes and outcomes as well as deficits in provision of stroke services and processes as recommended by WHO. These included a low number of specialist clinicians including neurologists, neuro-radiographers and neurosurgeons. The high prevalence of stroke suggests attention and resources need to focus on primary and secondary prevention. Awareness of stroke is low, especially in terms of what a stroke is, the signs/symptoms and how to minimize risk of stroke [ 25 ]. Barriers exist, such as lack of healthcare resources, maldistribution of health facilities, inadequate training on stroke treatment among health care workers, poor stroke awareness, insufficient government support and limited health insurance coverage [ 22 ].

The scoping review also highlighted areas where further work is needed, for example, descriptions and research into the frequency, intensity, and content of rehabilitation services especially in the community setting and the outcome measures used to monitor recovery and impairment. PARM published stroke rehabilitation clinical practice guidelines in 2012, which incorporated an innovative approach to contextualize Western clinical practice guidelines for stroke care to the Philippines [ 42 ]. Unfortunately, availability and equitable access to evidence-based rehabilitation for people with stroke in the Philippines pose significant challenges because of multiple factors impacting the country (e.g., geographical, social, personal, environmental, educational, economic, workforce) [ 25 , 40 , 43 ].

The number of stroke survivors with disability has not been reported previously, thus, the extent and burden of stroke from acute to chronic is unknown. The recent introduction of a national stroke registry across public and private facilities may provide some of this data [ 82 ]. The project started in 2021 and captures data on people hospitalized for transient ischemic attack or stroke in the Philippines. National stroke registries have been identified as a pragmatic solution to reduce the global burden of stroke [ 83 ] through surveillance of incidence, prevalence, and outcomes (e.g., death, disability) of, and quality of care for, stroke, and prevalence of risk factors. For the Philippine government to know the full impact and burden of stroke nationally, identify areas for improvement and make meaningful changes for the benefit of Filipinos, the registry would need to be compulsory for all public and private facilities and include out of hospital data. This will require information technology, trained workforces for data capture, monitoring and sharing, as well as governance and funding [ 83 ].

This scoping review has generated a better understanding of the published evidence focusing on availability of stroke services in the Philippines, as well as the existing gaps through the lens of Donabedian’s Structure , Process and Outcome framework. The findings have helped to inform a wider investigation of current stroke service utilization conducted using survey and interview methods with stroke survivors, carers and key stakeholders in the Philippines, and drive forward local, regional and national policy and service changes.

Conclusions

This scoping review describes the existing evidence-based relating to structure, processes and outcomes of stroke services for adults within the Philippines. The review revealed limited information in certain areas, such as the impact of stroke on functional ability, participation in everyday life, and quality of life; the content and intensity of rehabilitation both in the hospital or community setting; and the outcome measures used to evaluate clinical practice. Developments in stroke services have been identified however, a wide gap exists between the availability of stroke services and the high burden of stroke in the Philippines. Strategies are critical to address the identified gaps as a precursor to improving stroke outcomes and reducing burden. Potential solutions identified within the review will require a comprehensive approach from healthcare policymakers to focus on stroke awareness programs, primary and secondary prevention, establishing and monitoring of stroke protocols and pathways, implementation of a compulsory national stroke registry, use of TeleRehab, TeleMedicine and mobile stroke units and improve access to and availability of both hospital- and community-based stroke rehabilitation. Furthermore, changes in PhilHealth coverage and universal credit to minimize catastrophic out-of-pocket costs.

Limitations

Although a comprehensive search was undertaken, data were taken from a limited number of located published studies on stroke in the Philippines. This, together with data from databases and grey literature, may not reflect the current state of stroke services in the country.

Availability of data and materials

Not applicable.

Data availability

No datasets were generated or analysed during the current study.

Navarro JC, Baroque AC, Lokin JK, Venketasubramanian N. The real stroke burden in the Philippines. Int J Stroke. 2014;9(5):640–1.

Article PubMed Google Scholar

Philippines TSSot. Phillipines: stroke 2024. Available from: https://www.strokesocietyphilippines.org/philippines-stroke/#:~:text=Stroke%20is%20the%20Philippines'%20second,or%2014.12%25%20of%20total%20deaths .

Banaag MS, Dayrit MM, Mendoza RU. Health Inequity in the Philippines. In: Batabyal A, Higano Y, Nijkamp P (eds). Disease, Human Health, and Regional Growth and Development in Asia. New Frontiers in Regional Science: Asian Perspectives, vol 38. Singapore: Springer; 2019.

Hodge A, Firth S, Bermejo R, Zeck W, Jimenez-Soto E. Utilisation of health services and the poor: deconstructing wealth-based differences in facility-based delivering in teh Philippines. BMC Public Health. 2016;16:1–12.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.

Article Google Scholar

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.

Article PubMed PubMed Central Google Scholar

Donabedian A. The quality of care. How can it be assessed? JAMA. 1988;260(12):1743–8.

Article CAS PubMed Google Scholar

McDonald KM, Sundaram V, Bravata DM, Lewis R, Lin N, Kraft SA, et al. Closing the quality gap: a critical analysis of quality improvement strategies. Tech Rev. 2007;7(9).

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

McGlade B, Mendoza VE. Philippines CBR manual: an inclusive development strategy. Philippines: CBM-CBR Coordinating office; 2009.

Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5(210). https://doi.org/10.1186/s13643-016-0384-4 .

Baliguas B. Adherence to the clinical practice guidelines of the stroke society of the Philippines in the management of ischemic stroke in young adults admitted in 3 tertiary hospitals in Bacolod City, Philippines from May to October 2010. Neurology. 2018;90(15).

Barcelon EA, Moll MAKDN, Serondo DJ, Collantes MEV. Validation of the Filipino version of national institute of health stroke scale. Clinical Neurology. 2016;56:S379.

Google Scholar

Baticulon RE, Lucena LLN, Gimenez MLA, Sabalza MN, Soriano JA. The Neurosurgical Workforce of the Philippines. Neurosurgery. 2024;94(1):202–11. https://doi.org/10.1227/neu.0000000000002630 .

Berroya RM. Incidence of symptomatic intracerebral hemorrhage after thrombolysis for acute ischemic stroke at St. Luke’s Medical Center-Global City from January 2010 to February 2017. J Neurol Sci. 2010;2017(381):398–9.

Carcel C, Espiritu-Picar R. Circadian variation of ischemic and hemorrhagic strokes in adults at a tertiary hospital: a retrospective study. J Neurol Sci. 2009;285:S174.

Cayco CS, Gorgon EJR, Lazaro RT. Proprioceptive neuromuscular facilitation to improve motor outcomes in older adults with chronic stroke. Neurosciences (Riyadh). 2019;24(1):53–60.

Co COC, Yu JRT, Macrohon-Valdez MC, Laxamana LC, De Guzman VPE, Berroya-Moreno RMM, et al. Acute stroke care algorithm in a private tertiary hospital in the Philippines during the COVID-19 pandemic: a third world country experience. J Stroke Cerebrovasc Dis. 2020;29(9):105059.

Co COC, Yu JRT, Laxamana LC, David-Ona DIA. Intravenous thrombolysis for stroke in a COVID-19 positive Filipino patient, a case report. J Clin Neurosci. 2020;77:234–6.

Article CAS PubMed PubMed Central Google Scholar

Collantes ME. Evaluation of change in stroke care in the Philippines using RES-Q data. Eur Stroke J. 2019;4:318.

Collantes ME. Improving stroke systems of care in lmic: Philippines. Int J Stroke. 2021;16(2):4.

Collantes ME, Navarro J, Belen A, Gan R. Stroke systems of care in the Philippines: addressing gaps and developing strategies. Front Neurol. 2022;13:1046351.

Collantes MEV, Yves Miel H, Zuñiga Uezono DR. Incidence and prevalence of stroke and its risk factors in the Philippines: a systematic review. Acta Medica Philippina. 2022;56:26–34.

Collantes MV, Zuniga YH, Granada CN, Uezono DR, De Castillo LC, Enriquez CG, et al. Current state of stroke care in the Philippines. Front Neurol. 2021;12:665086.

Constantino GA, Soliven JA. Points of in-hospital delays in thrombolytic therapy among patients with acute ischemic stroke: a single center 5-year retrospective study. Neurology. 2020;94(15). https://doi.org/10.1212/WNL.94.15_supplement.2901 .

Constantino GAA, Señga MMA, Soliven JAR, Jocson VED. Emerging Utility of Endovascular Thrombectomy in the Philippines: A Single-center Clinical Experience. Acta Med Philipp [Internet]. 2023;57(5). Available from: https://actamedicaphilippina.upm.edu.ph/index.php/acta/article/view/5113 . [cited 2024 Aug 21].

Dans AL, Punzalan FE, Villaruz MV. National Nutrition and Health Survey (NNHeS): atherosclerosis-related diseases and risk factors. Philipp J Intern Med. 2005;43:103–15.

De Castillo LL, Collantes ME. Thrombolysis for stroke at the Philippine general hospital: a descriptive analysis. Cerebrovasc Dis. 2019;48:54.

de Castillo LLC, Diestro JDB, Tuazon CAM, Sy MCC, Añonuevo JC, San Jose MCZ. J Stroke Cerebrovasc Dis. 2021;30(7):105831.

Delfino JPM, Carandang-Chacon CA. Comparison of acute ischemic stroke care quality before and during the COVID-19 pandemic in a private tertiary hospital in metro Manila, Philippines. Neurol Asia. 2023;28(1):13–7.

Department of Health. Department of Health Administrative Order 2011-0003. 2011. [Accessed online: 12/2022], from the Philippine Department of Health].

Department of Health. The national policy framework on the prevention, control and management of acute stroke in the Philippines. 2020.

Diestro JDB, Omar AT, Sarmiento RJC, Enriquez CAG, Castillo LLC, Ho BL, et al. Cost of hospitalization for stroke in a low-middle-income country: Findings from a public tertiary hospital in the Philippines. Int J Stroke. 2021;16(1):39–42.

Duenas M, Ranoa G, Benjamin VS. Assessment of post-stroke caregivers’ burden through the modified caregivers strain index (MCSI) in a tertiary center in the Philippines: a cross-sectional study. Cerebrovasc Dis. 2019;48:56–7.

Duya JE, Hernandez K, San Jose MC. The evolving clinical and echocardiographic profile of patients admitted for acute cardioembolic stroke at a Tertiary Hospital in the Philippines. J Hong Kong Coll Cardiol. 2019;27(1):58.

Espiritu AI, San Jose MCZ. A call for a stroke referral network between primary care and stroke-ready hospitals in the philippines: a narrative review. Neurologist. 2021;26(6):253–60.

Gambito ED, Gonzalez-Suarez CB, Grimmer KA, Valdecañas CM, Dizon JM, Beredo ME, et al. Updating contextualized clinical practice guidelines on stroke rehabilitation and low back pain management using a novel assessment framework that standardizes decisions. BMC Res Notes. 2015;8:643.

Gelisanga MA, Gorgon EJ. Upright motor control test: interrater reliability, retest reliability, and concurrent validity in adults with subacute stroke. Eur Stroke J. 2017;2(1):357–8.

Gonzalez-Suarez C, Grimmer K, Alipio I, Anota-Canencia EG, Santos-Carpio ML, Dizon JM, et al. Stroke rehabilitation in the Philippines: an audit study. Disabil CBR Inclusive Develop. 2015;26(3):44–67.

Gonzalez-Suarez CB, Grimmer K, Cabrera JTC, Alipio IP, Anota-Canencia EGG, Santos-Carpio MLP, et al. Predictors of medical complications in stroke patients confined in hospitals with rehabilitation facilities: a Filipino audit of practice. Neurology Asia. 2018;23(3):199–208.

Gonzalez-Suarez CB, Grimmer-Somers K, Margarita Dizon J, King E, Lorenzo S, Valdecanas C, et al. Contextualizing Western guidelines for stroke and low back pain to a developing country (Philippines): an innovative approach to putting evidence into practice efficiently. J Healthc Leadersh. 2012;4:141–56.

Gonzalez-Suarez CB, Margarita J, Dizon R, Grimmer K, Estrada MS, Uyehara ED, et al. Implementation of recommendations from the Philippine Academy of Rehabilitation Medicine's Stroke Rehabilitation Guideline: a plan of action. Clin Audit. 2013;5:77–89.

Ignacio KHD, Diestro JDB, Medrano JMM, Salabi SKU, Logronio AJ, Factor SJV, et al. Depression and anxiety after stroke in young adult Filipinos. J Stroke Cerebrovasc Dis. 2022;31(2):106232.

Inting K, Canete MT. Ischemic stroke subtypes: a comparison between causative and phenotypic classifications in a tertiary hospital in the Philippines. Int J Stroke. 2021;16(2):28.

Jaca PKM, Chacon CAC, Alvarez RM. Clinical characteristics of cerebrovascular disease with COVID-19: a single-center study in Manila. Philippines Neurology Asia. 2021;26(1):15–25.

Jamora RDG, Corral EV, Ang MA, Epifania M, Collantes V, Gan R. Stroke recurrence among Filipino patients taking aspirin for first-ever non-cardioembolic ischemic stroke. Neurol Clin Neurosci. 2017;5:1–5.

Jamora RDG, Prado MB Jr, Anlacan VMM, Sy MCC, Espiritu AI. Incidence and risk factors for stroke in patients with COVID-19 in the Philippines: an analysis of 10,881 cases. J Stroke Cerebrovasc Dis. 2022;31(11).

Juangco DN, Mariano GS. Endovascular therapy for acute ischemic stroke: a review of cases and outcomes from a primary stroke center (a 5-year retrospective study). Cerebrovasc Dis. 2016;41:54.

Leochico CFD, Austria EMV, Gelisanga MAP, Ignacio SD, Mojica JAP. Home-based telerehabilitation for community-dwelling persons with stroke during the COVID-19 pandemic: a pilot study. J Rehabil Med. 2023;55:jrm4405.

Loo KW, Gan SH. Burden of stroke in the Philippines. Int J Stroke. 2013;8(2):131–4.

Mansouri A, Ku JC, Khu KJ, Mahmud MR, Sedney C, Ammar A, et al. Exploratory analysis into reasonable timeframes for the provision of neurosurgical care in low- and middle-income countries. World Neurosurg. 2018;117:e679–91.

Mendoza RA. The clinical profile and treatment outcome of acute ischemic stroke patients who underwent thrombolysis with recombinant tissue plasminogen activator therapy, Philippine experience: a retrospective study. J Neurol Sci. 2009;285:S85–6.

Navarro J. Prevalence of stroke: a community survey. Philipp J Neurol. 2005;9(2):11–5.

Navarro JC, Venketasubramanian N. Stroke burden and services in the Philippines. Cerebrovasc Dis Extra. 2021;11(2):52–4.

Navarro JC, Baroque AC 2nd, Lokin JK. Stroke education in the Philippines. Int J Stroke. 2013;8 Suppl A100:114–5.

Navarro JC, Chen CL, Lee CF, Gan HH, Lao AY, Baroque AC, et al. Durability of the beneficial effect of MLC601 (NeuroAiD™) on functional recovery among stroke patients from the Philippines in the CHIMES and CHIMES-E studies. Int J Stroke. 2017;12(3):285–91.

Navarro JC, Escabillas C, Aquino A, Macrohon C, Belen A, Abbariao M, et al. Stroke units in the Philippines: an observational study. Int J Stroke. 2021;16(7):849–54.

Navarro JC, San Jose MC, Collantes E, Macrohon-Valdez MC, Roxas A, Hivadan J, et al. Stroke thrombolysis in the Philippines. Neurol Asia. 2018;23(2):115.

Ng JC, Churojana A, Pongpech S, Vu LD, Sadikin C, Mahadevan J, et al. Current state of acute stroke care in Southeast Asian countries. Interv Neuroradiol. 2019;25(3):291–6.

Ocampo FF, De Leon-Gacrama FRG, Cuanang JR, Navarro JC. Profile of stroke mimics in a tertiary medical center in the Philippines. Neurol Asia. 2021;26(1):35–9.

Pascua R, Hiyadan JH. Outcome of decompressive hemicraniectomy without evacuation of hematoma in supratentorial intracerebral hemorrhage in a tertiary government hospital in the Philippines: a retrospective study. Eur Stroke J. 2023;8(2):586.

Prado M, Jamora RD, Charmaine Sy M, Anlacan M, Espiritu A. Determinants and Outcomes of Cerebrovascular Disease in Patients with COVID19 in the Philippines: An Analysis of 10881 Cases. Neurology. 2022;98(18). https://doi.org/10.1212/WNL.98.18_supplement.2076 .

Qua CV, Tiqui V, Villatima NE, Perales DJ, Rubio SM, Santos ER, et al. A predictive assessment of early neurological deterioration among Filipino acute ischemic stroke patients utilizing hematological, lipid profile, and metabolic parameters in a tertiary hospital in Pampanga. Philippines Cerebrovasc Dis. 2022;51:101.

Que DL, Cuanang J, San Jose MC. Clinical profile, management and outcomes of patients with cerebralvenous thrombosis in atertiary hospital in the Philippines. Int J Stroke. 2020;15(1):511.

Quiles LEP, Diamante PAB, Pascual JLV. Impact of the COVID-19 pandemic in the acute stroke admissions and outcomes in a Philippine Tertiary Hospital. Cerebrovasc Dis Extra. 2022;12(2):76–84.

Roxas AA. The RIFASAF project: a case-control study on risk factors for stroke among Filipinos. Philippine J Neurol. 2002;6(1):1–7.

Roxas AAC, Carabal-Handumon J. Knowledge and perceptions among the barangay health workers in Plaridel, Misamis Occidental. Philipp J Neurol. 2002;6(1):44.

Sasikumar S, Bengzon Diestro JD. Global & community health: acute ischemic stroke in Toronto and Manila: bridging the gap. Neurology. 2020;95(13):604–6.

Senga MM, Reyes JPB. Cerebral venous thrombosis in a single center tertiary hospital of a South East Asian country (CVSTS study)-a retrospective study on the clinical profiles of patients with cerebral venous thrombosis. Neurology. 2019;92(15). https://doi.org/10.1212/WNL.92.15_supplement.P5.3-011 .

Sese LVC, Guillermo MCL. Strengthening stroke prevention and awareness in the Philippines: a conceptual framework. Front Neurol. 2023;14:1258821.

Suwanwela NC, Chen CLH, Lee CF, Young SH, Tay SS, Umapathi T, et al. Effect of combined treatment with MLC601 (NeuroAiDTM) and rehabilitation on post-stroke recovery: the CHIMES and CHIMES-E studies. Cerebrovasc Dis. 2018;46(1–2):82–8.

Talamera AF, Franco DS. Validation study of Siriraj stroke score in Southern Philippines. Cerebrovasc Dis. 2011;32:9.

Tan A, Navarro J. Outcomes and quality of care outcome of patients with primary intracerebral hemorrhage in a single center in the philippines. Int J Stroke. 2014;9:269.

Tangcuangco NC, Bitanga ES, Roxas AA, Pascual JL, Saniel E, Reyes JP, et al. Intravenous recombinant tissue plasminogen activator (IV-rtPA) use in acute ischemic stroke in a private tertiary hospital: a Philippine setting. Int J Stroke. 2010;5:107.

Tsang ACO, Yang IH, Orru E, Nguyen QA, Pamatmat RV, Medhi G, et al. Overview of endovascular thrombectomy accessibility gap for acute ischemic stroke in Asia: a multi-national survey. Int J Stroke. 2020;15(5):516–20.

Vatanagul J, Cantero-Auguis C. Awareness on acute stroke management among family medicine and internal medicine residents in Metro Cebu. Philippines J Neurol Sci. 2015;357:e418–9.

Vatanagul J, Rulona IA. The incidence of post-stroke depression in a tertiary hospital in Cebu City, Philippines. J Neurol Sci. 2015;357:e419.

Vatanagul JAS, Rulona IA, Belonguel NJ. Cerebral venous thrombosis (CVST): study of four Filipino patients and literature review. Cerebrovasc Dis. 2013;36:81.

Venketasubramanian N, Yoon BW, Pandian J, Navarro JC. Stroke epidemiology in south, east, and south-east Asia: a review. J Stroke. 2017;19(3):286–94.

Yu RF, San Jose MC, Manzanilla BM, Oris MY, Gan R. Sources and reasons for delays in the care of acute stroke patients. J Neurol Sci. 2002;199(1–2):49–54.

Philippine Neurological Association One Database - Stroke DsSMG. Multicentre collection of uniform data on patients hospitalised for transient ischaemic attack or stroke in the Philippines: the Philippine Neurological Association One Database-Stroke (PNA1DB-Stroke) protocol. BMJ Open. 2022;12(5):54.

Feigin VL, Owolabi MO, Group WSOLNCSC. Pragmatic solutions to reduce the global burden of stroke: a world stroke organization-lancet neurology commission. Lancet Neurol. 2023;22(12):1160–206.

Download references

Acknowledgements

We acknowledge the TULAY collaborators: Dr Roy Francis Navea, Dr Myrna Estrada, Dr Elda Grace Anota, Dr Maria Mercedes Barba, Dr June Ann De Vera, Dr Maria Elena Tan, Dr Sarah Buckingham and Professor Fiona Jones. We are grateful to Lance de Jesus and Dr Annah Teves, Research Assistants on the TULAY project, for their contribution to some of the data extraction.

This research was funded by the NIHR Global Health Policy and Systems Research Programme (Award ID: NIHR150244) in association with UK aid from the UK Government to support global health research. The views expressed in this publication are those of the authors and not necessarily those of the NIHR or the UK’s Department of Health and Social Care.

Author information

Authors and affiliations.

Faculty of Health, Intercity Place, University of Plymouth, Plymouth, Devon, PL4 6AB, UK

Angela Logan, Bridie Kent, Aira Ong & Jonathan Marsden

Royal Devon University Healthcare NHS Foundation Trust, William Wright House, Barrack Road, Exeter, Devon, EX2 5DW, UK

Angela Logan

De La Salle University-Evelyn D. Ang Institute of Biomedical Engineering and Health Technologies, 2401 Taft Avenue, Malate, Manila, 1004, Philippines

Lorraine Faeldon

The University of Plymouth Centre for Innovations in Health and Social Care: A JBI Centre of Excellence, Faculty of Health, Intercity Place, University of Plymouth, Plymouth, Devon, PL4 6AB, UK

Bridie Kent

You can also search for this author in PubMed Google Scholar

Contributions

Conceptualisation, methodology and setting search terms, AL, LF, AO, JM, BK. Searches and screening, AL, JM, LF, AO. Data extraction, AL, LF, AO, JM, LdJ, AT. Original draft preparation, AL, JM. All authors provided substantive intellectual and editorial revisions and approved the final manuscript.

Corresponding author

Correspondence to Angela Logan .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Logan, A., Faeldon, L., Kent, B. et al. A scoping review of stroke services within the Philippines. BMC Health Serv Res 24 , 1006 (2024). https://doi.org/10.1186/s12913-024-11334-z

Download citation

Received : 20 March 2024

Accepted : 22 July 2024

Published : 30 August 2024

DOI : https://doi.org/10.1186/s12913-024-11334-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Stroke care
Low- middle-income countries
Developing countries
Philippines

BMC Health Services Research

ISSN: 1472-6963

General enquiries: [email protected]

Open access
Published: 30 August 2024

Impact of partner alcohol use on intimate partner violence among reproductive-age women in East Africa Demographic and Health Survey: propensity score matching

Mamaru Melkam 1 ,
Bezawit Melak Fente 2 ,
Yohannes Mekuria Negussie 3 ,
Zufan Alamrie Asmare 4 ,
Hiwot Altaye Asebe 5 ,
Beminate Lemma Seifu 5 ,
Meklit Melaku Bezie 6 &
Angwach Abrham Asnake 7

BMC Public Health volume 24 , Article number: 2365 ( 2024 ) Cite this article

Metrics details

Introduction

Intimate Partner Violence (IPV) is the most prevalent form of violence against women globally and is more prevalent than rape or other violent attacks by strangers. Different observational studies have established a strong positive association between alcohol use and intimate partner violence. Even though there are a lot of studies that show the association between partner alcohol use and intimate partner violence limited studies were conducted that show the direct causative relations of partner alcohol use and IPV among reproductive-age women in East Africa. Therefore, this study aimed to determine the effect of partner alcohol use on intimate partner violence in East Africa’s recent Demographic and Health Survey (DHS) data with Propensity Score Matching (PSM).

Community-based cross-sectional study design with a propensity score matching was used from the East African countries’ DHS data. A total of the weighted sample size of 72,554 reproductive-age women was used for this study. Propensity score matching analysis was conducted to determine the causal relation between partner alcohol use and intimate partner violence. Intimate partner violence was the outcome variable and partner alcohol use was the treatment variable. Propensity score matching was carried out through Stata software by using psmatch2 of the logit-based model. The assumption of common support was verified and achieved. Mantel-Haenszel boundaries have been used to investigate the possibility of hidden bias in the outcome.

The prevalence of partner alcohol use and intimate partner violence from East African countries was 37.94 with a CI of (37.58%, 38.29%) and 41.45% with a CI (41.09%, 41.80%) respectively. Partner alcohol use contributed to a 2.78% increase in intimate partner violence according to the estimated average treatment on treated values in the treated and control groups were 59.41% and 31.51%, respectively. Ultimately, it was found that among all research participants, the average effect on the population as a whole was 25.33%.

We conclude that partner alcohol use has a direct cause for intimate partner violence. Therefore, controlling partner alcohol consumption can reduce the burden of intimate partner violence.

Peer Review reports

The World Health Organization (WHO) defines intimate partner violence as the deliberate act of an intimate partner or former spouse that results in sexual misconduct, severe physical harm, emotional abuse, or dominating activities [ 1 ]. Intimate partner violence is the most prevalent type of violence against women, with major health consequences, and is more likely to occur in homes rather than on street level. Intimate partner violence increases the risk of gynecological, neurological, and stressful problems for women [ 2 ]. Alcohol is the most popular beverage in the world and a fluid that includes ethanol. Worldwide commonly men drinking alcohol is associated with numerous misconducts, including violence against their intimate partners [ 3 ]. Alcohol’s psychophysiological effects are considered to directly increase the risk of criminalizing IPV in those who consume alcohol [ 4 ]. The potential habit of alcohol addiction has made it difficult to determine whether there is a causal relationship between alcohol abuse and IPV [ 5 ].

Alcohol consumption is one of the common and well-established risk factors for intimate partner violence. Although different measures have been made to lessen intimate partner violence it remains a significant public health issue that requires additional work to address [ 6 ]. It’s unclear how the etiological theories put up to explain the connection between alcohol use and IPV have been tested in earlier studies conducted in low and middle-income countries [ 5 ]. Regardless of the consequences of alcohol consumption, some drinkers may purposefully act violently or aggressively toward their spouse in the hopes that their actions will be accepted as they were under the influence of alcohol drink [ 7 ].

The prevalence of IPV among reproductive-age women in the world including Africa varies greatly [ 2 ]. IPV is a serious public health issue and an attack on women’s human rights; globally, nearly one-third (27%) of women between the ages of 15 and 49 who were in a relationship have encountered sexual or severe physical assault at the expense of their intimate partner [ 8 , 9 ]. One of the most prevalent forms of violence against women is intimate partner violence [ 10 ]. The prevalence of partner alcohol use and intimate partner violence in Sub-Saharan Africa ranges from 3 to 62% and 11–60% respectively. In other studies in Africa, the burden of partner alcohol use was 36.3% with a prevalence of IPV (9.7–25.0%) [ 11 ].

There are several factors in the previous observational study which show the association between partner alcohol use and intimate partner violence [ 12 , 13 ]. The association of factors between alcohol use and intimate partner violence in developing countries was confounded by a wide range of factors that exist at the individual and community level variables [ 4 ]. The variables associated with IPV were incorporated: sex of male household head, age, occupation, educational status, marital status, mass media exposure, wealth status, and number of children [ 12 , 14 , 15 ].

World Health Organization suggested that primary prevention strategies aimed at minimizing alcohol-related harm could also potentially minimize IPV even though drinking alcohol can occur without IPV and IPV can occur without alcohol consumption [ 16 ]. Different literature evidence that it is very difficult to determine the degree of the associations between substance consumption and intimate partner violence [ 17 ]. The exact causal relation is difficult to determine by observational study due to the presence of other associated factors. Besides the variable observed, there are also unobserved variables and biases that prohibit the exact causal relations between alcohol use and intimate partner violence. Propensity Score Matching (PSM) analysis is the best technique to avoid bias through matching partner alcohol use (treatment group) and partner, not alcohol use (control group) among reproductive-age women with similar exposure to intimate partner violence. According to our best knowledge, there are no studies in East Africa that show the effect of partner alcohol use on intimate partner violence among reproductive-age women. Therefore, this study aimed to determine the impact of partner alcohol use on intimate partner violence in East Africa’s recent DHS data with propensity score matching.

Method and material

Study design and area.

A community-based cross-sectional study was employed on the recent Demography and Health Survey (DHS) data of 12 East African countries (Burundi, Comoros, Ethiopia, Kenya, Madagascar, Malawi, Mozambique, Rwanda, Tanzania, Uganda, Zambia, and Zimbabwe) from 2016 to 2024. East African countries’ DHS data included fertility, reproductive health, maternal and child health, mortality, nutrition, and self-reported health reports. The DHS incorporated datasets are on men, women, children, births, and households for this survey and we have used the women’s data for this secondary data analysis. Reproductive-age women between the ages of 15 to 49 were included in this secondary data analysis. The total weighted sample size for this study was 72,554 with 1692 clusters. The detailed data can be accessed comprehensively by clicking on the official link http://www.dhsprogram.com/ [ 18 ].

Operational definitions

The revised version of the domestic violence questionnaire module was used. Study participants who have experienced intimate partner violence in their lifetime were considered in this study. The module of questions on GBV was administered following the World Health Organization’s guidelines on the ethical collection of information on gender-based violence (WHO 2001). From a total of nine revised versions of the domestic violence questionnaire scoring one and above were considered as having IPV.

Alcohol use: It was assessed by the simple question of alcohol drinking every day. Study participants whose husband drinks alcohol on an everyday basis and with harmful consequences even though the specific amount and frequency, it is the limitation of the study.

Outcome and treatment variables

Partner alcohol use is measured by a single question; did your husband/partner drink alcohol every day? The dependent variable for this study is intimate partner violence which includes severe physical violence, sexual violence, and emotional violence. It was measured by the following questions and participants who have said yes to at least one question are considered as they have IPV.

Severe physical violence

Ever been kicked or dragged by your husband?

Ever been strangled or burned by a husband?

Ever been threatened with a knife, gun, or another weapon?

Sexual violence

Ever been physically forced to have unwanted sex by your husband?

Ever been forced to do other sexual acts by your husband?

Ever been forced to perform sexual acts respondent didn’t want to?

Emotional violence

Ever been humiliated by your husband?

Ever been threatened with harm by your husband?

Ever been insulted or made to feel bad by your husband?

Data management and statistical analysis

When randomization was not an option, propensity score matching was frequently used to ascertain the effects of treatment in experimental designs. Because of bias caused by an imbalance in observable factors that modifies the causal influence of experience, study participants were randomly assigned to one of the groups. Adjust and rectify group inequality using a balancing score to fix the imbalance between the groups using PSM while confounding factors can be identified. The balance score indicates that the treatment group should not affect the observed variable which is partner alcohol use. After propensity score adjustments for the observed covariates, the difference in outcomes between those who experienced intimate partner violence and those who did not offer an objective measure of the impact of partner alcohol use on intimate partner violence becomes equal. The propensity score which always ranges from 0 to 1 is a conditional likelihood of receiving treatment (partner alcohol use). A higher propensity score indicates that women whose partners drink alcohol. The treatment variables of interest in propensity score matching need to be dichotomous. The imbalance of covariates between the treatment and control groups is assessed using a t-test for continuous factors and a chi-square for categorical components.

Based on the association between the outcome and treatment variables, three separate results were obtained from the observed covariant. However, the only one that can be added is PSM. The likelihood that a woman might have partner alcohol use is reduced to a propensity score for each woman based on the variables selected. A propensity score for each participant is generated with the selected confounders [ 19 , 20 , 21 ]. PSM approach was used for those women with partner alcohol use that wasn’t distributed randomly between the two groups and might be considerably impacted by both observable and non-observable factors. PSM covariates were incorporated as they have a strong association with partner alcohol use and intimate partner violence, including socio-demographic and behavioral factors. Variables included before PSM (sex of household head, age, education, currently working, wealth status, partner alcohol use, and husband education) had significant differences with a p -value of less than 0.05 with IPV among those who have alcohol user partners and do not have. The variables mentioned above showed no significant difference for IPV while participants with and without partner alcohol use were matched with a p -value of greater than 0.05. This suggests that PSM dramatically reduced the group’s observed variable difference.

The most widely accepted PSM hypotheses are a selection of unobservable variables and common support that have been evaluated statistically and graphically. Throughout the study, the common support option was taken into account to limit the balance of propensity to mothers with treatment (partner alcohol use) whose propensity score for IPV was within the ranges of propensity scores for controls. We tested two types of matching methods: nearest neighbor matching with and without replacement and radius matching with calipers ranging from 0.01 to 0.05. Stata psmatch2 was used to calculate the Average Treatment Effect for treated (ATT), Average Treatment effect on Untreated (ATU), and Average Treatment effect for the whole population (ATE) for the matching technique that produced the most effective matches. Standard supported option was also used to generate higher-quality matches. The basis for assessing the quality of matching was the balance of the variables between the treated and control groups. To determine the degree of matching, the standardized bias before and after matching was calculated. The difference in percentages is used to compute this bias.

The percentage of the square root of the average sample variances in both groups was used to check the percentage difference between the sample means in the matched control and treatment groups. Although, there is no hard and fast rule on the degree of standardized difference to indicate an imbalance variation of less than 10% is considered a low variation. The pseudo-R2 and likelihood ratio tests were used to examine the joint importance of all the covariates from the logit estimation of the conditional treatment probability before and after matching. A sensitivity analysis was used to evaluate the PSM estimations’ reliability [ 22 ]. Because the outcome variable was binary, the Mantel-Haenzel (MH) test statistic was used to assess whether the PSM estimates were sensitive to the hidden bias [ 23 ]. The gamma coefficient quantifies the unobserved confounding or hidden bias that affects how the treatment is allocated to the treated and control groups. Using the mhbounds STATA command, the gamma value ranges from 1 to 2 with a 0.05 increment.

Study participant descriptive characteristics

A total of 72,554 reproductive-age women aged from 15 to 49 were used from the East African countries DHS data. The prevalence of partner alcohol use and intimate partner violence from East African countries was 37.94 with a CI of (37.58%, 38.29%) and 41.45% with a CI (41.09%, 41.80%) respectively. The characteristics of study participants before matching were described according to partner alcohol use (Table 1 ). From this propensity score matching analysis women’s age, partner age, maternal occupation, media exposure, sex household, wealth index, paternal education, and residence were significantly associated with a p -value of less than 0.05 with partner alcohol use before matching (Table 2 ).

Estimations of propensity score

The estimations of the association’s orientation, power, and significance aligned with the findings of other researchers (Table 2 ). The minimum variability with mean propensity score among the intervention and control groups was 1.44. The range of propensity scores varied from 0.08 to 1.24 which showed the common support assumption was satisfied. Reproductive-age women whose propensity scores fell below the range of common support were dropped from either the treatment or control groups.

Impact of partner alcohol use on intimate partner violence

The unmatched estimate indicates that women who have partner alcohol users are 2.89% more likely to have IPV than women who have not. The nearest neighbor matching had the best matching quality with a caliper width of 0.01. IPV increased by 2.78% as a result of partner alcohol consumption, according to an estimated average treatment on treated values of 59.41% in the treated and 31.51% in the control group. Similarly, the estimated average treatment effect on untreated values in the control group and treated group was 30.46% and 54.24%, respectively. This finding indicated that if the women who hadn’t partner alcohol use had been encountered with partner who uses alcohol the chance of developing IPV would have increased by 23.77%. Ultimately, it was found that among all research participants, the average effect on the population as a whole was 25.33% (Table 3 ).

Quality of matching

Common support.

Only one woman was eliminated because of off-support (Table 4 ). The propensity score distributions for both groups are almost identical when plotted on PSM after matching (Fig. 1 ). The significant overlap between the treatment and control groups’ features validates the common support assumption.

Propensity score histogram by treatment status (partner alcohol use)

Balancing test

The test’s significance level was established and the t-test was utilized to evaluate the difference between the matched and unmatched pairs. Almost all factors displayed no significant mean difference following matching, despite a significant mean difference across all covariates (Table 5 ). This proved that for every variable in the model, the treated and control groups were appropriately balanced.

Standardized bias

The pstest’s mean and median biases considerably lowered once the intervention and control groups were matched. The mean absolute bias in the unpaired sample decreased from 15.5 to 1.1% after the treated and control groups were matched. This is less than the 5% threshold and shows that the model’s quality matching has improved. The median bias decreased from 13.8% in the unmatched to 0.3% after matching (Table 5 ).

Model significance

The overall significance of the model was assessed using the LR and pseudo R2 tests. The pseudo-R2 was less than 0.001 and the LR-chi2 test had become negligible ( p = 1.0), suggesting that there was no systematic variation in the covariate distribution between the treated and control groups (Table 6 ).

Sensitivity analysis

The Mantel-Haenszel finding indicated that the overestimation of partner alcohol use effect on IPV was not significant at 5% of the significance level. However, the statistical significance of the underestimating of partner alcohol use impact was established at a 5% level of significance. As gamma increases, the probability of underestimating the effect of partner alcohol use on IPV increases, suggesting a reduction in the possibility of heterogeneity due to unobserved factors (Table 7 ).

The main objective of this study was to determine the relationships between lifelong experiences of partner alcohol use and intimate partner violence among reproductive-age women. Secondary data analysis was conducted from the recent East African countries’ DHS data.

Based on PSM approach, partner alcohol use contributed to a 2.78% increase in intimate partner violence. The estimated average treatment on treated values in the treated and control groups were 59.41% and 31.51%, respectively. Comparably, the treated group’s estimated average treatment effect on untreated values was 54.24%, while the control group’s estimated average treatment effect was 30.46%. According to this research, the number of women experiencing IPV would have increased by 23.77% if they had met partner alcohol use instead of none. In the end, it was discovered that the average effect on the population as a whole for all research participants was 25.33%. This finding is comparable with other propensity score matching analyses that show the impact of partner alcohol use on IPV [ 4 ].

A significant positive association has been shown in a lot of studies between alcohol use and intimate partner violence. However, because people may misreport their alcohol abuse and because there may be reversed causality from IPV to alcohol abuse, it has been challenging to determine the causal relationship between alcohol abuse and IPV [ 5 ]. Additionally, there is a potential endogeneity issue, which suggests that those who are more likely to engage in excessive drinking are also more likely to engage in IPV due to an undetected third factor. Previous studies have demonstrated a correlation between colonization and alcohol consumption as a coping mechanism for being emotional of rage, avoidance, grief other factors [ 24 ].

These temporal correlations between frequent alcohol use by partners and IPV may be explained by several factors: Men who drink alcohol often may have poor judgment which makes it harder for them to recognize their fault and violation towards their intimate partners [ 25 ]. The effect of an intoxicated partner due to drinking alcohol was a great concern to cause intimate partner violence. Once men have been intoxicated after using alcohol their cognitive function entirely deteriorates which causes intimate partner violence among reproductive-age women in East Africa. The other justification for this association could be the effect of husbands who drink alcohol being easily tempered and aggressive toward their wives [ 26 ].

There is a frequent association between alcohol use and incidents of IPV among reproductive-age women. Although the idea that IPV causes alcohol usage cannot be completely ruled out, there is a lack of long-term data to support most previous studies [ 27 ]. This study determines the direct causative relation between partner alcohol use and intimate partner violence among reproductive-age women by 2.78%. This study estimates the correlation between alcohol consumption and IPV besides seeking to determine a causal relationship. According to DHS data, intimate partner violence is influenced by the partner’s alcohol consumption among women of reproductive age from East African countries.

Despite the presence of limitations, this study has several advantages. This is the first study to estimate bias through the determination of the causative relation of partner alcohol consumption on IPV into account using propensity score matching in East Africa. Nationally representative DHS data from 12 East African countries with a large sample size of 72,544 served as the foundation for this study and was used with a high response rate. The weakness of this study is the sensitivity of intimate partner violence results under-report their case. The variables that were observed provided the framework of the matching there might be a chance of the occurrence of residual confounding. Additionally, we have used DHS data with cross-sectional research that might have a social desirability and recall bias.

Conclusions and recommendations

Although it has been suggested that treating and preventing alcohol abuse is a good way to prevent IPV this guidance is not implemented widely in East African countries. These results highlight the necessity of using alcohol consumption reduction as a potential target for IPV prevention efforts and as a key correlate of IPV. These results imply that structural, macro-level actions may be able to reduce the causative association of alcohol use on IPV. When taken as a whole, these results emphasize the necessity of assessing multilayer intervention techniques to reduce or mitigate the causative association of alcohol use with intimate partner violence. Focusing on decreasing the partner’s alcohol consumption to mitigate the burden of intimate partner violence is our best recommendation.

Data availability

The DHS program repository contains the datasets that have been developed and/or assessed for this study, http://www.dhsprogram.com.

Abbreviations

Akaike Information Criteria

Adjusted Odd Ratio

Demographic Health Data

Confidence Interval

Intra-Class Correlation

Intimate Partner Violence

Median Odds Ratio

Proportional Change in Variance

World Health Organizations

Tsegaw M, Mulat B, Shitu K. Intimate partner violence and associated factors among reproductive age women in Liberia: further analysis of recent Liberian demographic and health survey. 2022.

Mulat B, Tsegaw M, Chilot D, Shitu K. Assessment of domestic violence and its associated factors among ever-married reproductive-age women in Cameroon: a cross-sectional survey. BMC Womens Health. 2022;22(1):397.

Article PubMed PubMed Central Google Scholar

Organization WH. Global status report on road safety 2018. World Health Organization; 2019.

Greene MC, Heise L, Musci RJ, Wirtz AL, Johnson R, Leoutsakos J-M, et al. Improving estimation of the association between alcohol use and intimate partner violence in low-income and middle-income countries. Inj Prev. 2021;27(3):221–6.

Article Google Scholar

Averett SL, Wang Y. Identifying the causal effect of alcohol abuse on the perpetration of intimate partner violence by men using a natural experiment. South Econ J. 2016;82(3):697–724.

Tessema ZT, Gebrie WM, Tesema GA, Alemneh TS, Teshale AB, Yeshaw Y, et al. Intimate partner violence and its associated factors among reproductive-age women in East Africa:-A generalized mixed effect robust poisson regression model. PLoS ONE. 2023;18(8):e0288917.

Article CAS PubMed PubMed Central Google Scholar

Wilson IM, Graham K, Taft A. Alcohol interventions, alcohol policy and intimate partner violence: a systematic review. BMC Public Health. 2014;14:1–11.

Article CAS Google Scholar

Bushman BJ. Aggression and violence. Psychology; 2016.

Tsegaw M, Mulat B, Shitu K. Intimate partner violence and associated factors among reproductive age women in Liberia: a cross-sectional study using a recent Liberian demographic and health survey. BMC Womens Health. 2022;22(1):238.

Organization WH. Violence against women. 2014.

Aboagye RG, Ahinkorah BO, Tengan CL, Salifu I, Acheampong HY, Seidu A-A. Partner alcohol consumption and intimate partner violence against women in sexual unions in sub-saharan Africa. PLoS ONE. 2022;17(12):e0278196.

Giff ST, Forkus SR, Massa AA, Brower JL, Jarnecke AM, Flanagan JC. Examining relationships among Alcohol Use Disorder, Child Caretaking, and intimate Partner violence in high-risk couples. J Family Violence. 2024:1–9.

Barker KM, Raj A. Understanding the roles of sport and alcohol use in adolescence on physical and sexual intimate partner violence perpetration in young adulthood: findings from a sex-stratified multilevel analysis. J Interpers Violence. 2022;37(13–14):NP10539–64.

Article PubMed Google Scholar

Greene MC, Kane J, Tol WA. Alcohol use and intimate partner violence among women and their partners in sub-saharan Africa. Global Mental Health. 2017;4:e13.

Cunradi CB, Mair C, Ponicki W, Remer L. Alcohol outlets, neighborhood characteristics, and intimate partner violence: ecological analysis of a California City. J Urb Health. 2011;88:191–200.

Organization WH. Preventing intimate partner and sexual violence against women: taking action and generating evidence. World Health Organization; 2010.

Eckhardt CI, Parrott DJ, Massa AA. Substance use and intimate partner violence perpetration. Handbook of interpersonal violence and abuse across the lifespan: a project of the National Partnership to end interpersonal violence across the Lifespan (NPEIV). Springer; 2021. pp. 2399–418.

Magadi M, Desta M. A multilevel analysis of the determinants and cross-national variations of HIV seropositivity in sub-saharan Africa: evidence from the DHS. Health Place. 2011;17(5):1067–83.

Deb S, Austin PC, Tu JV, Ko DT, Mazer CD, Kiss A, et al. A review of propensity-score methods and their use in cardiovascular research. Can J Cardiol. 2016;32(2):259–65.

Lanehart RE, Rodriguez de Gil P, Kim ES, Bellara AP, Kromrey JD, Lee RS, editors. Propensity score analysis and assessment of propensity score approaches using SAS procedures. Proceedings of the SAS Global Forum 2012 Conference; 2012: SAS Institute Inc Cary North Carolina.

Stone CA, Tang Y. Comparing propensity score methods in balancing covariates and recovering impact in small sample educational program evaluations. Practical Assess Res Evaluation. 2019;18(1):13.

Google Scholar

Becker SO, Caliendo M. Sensitivity analysis for average treatment effects. stata J. 2007;7(1):71–83.

Li L, Shen C, Wu AC, Li X. Propensity score-based sensitivity analysis method for uncontrolled confounding. Am J Epidemiol. 2011;174(3):345–53.

Les Whitbeck B, Chen X, Hoyt DR, Adams GW. Discrimination, historical loss and enculturation: culturally specific risk and resiliency factors for alcohol abuse among American indians. J Stud Alcohol. 2004;65(4):409–18.

Article CAS PubMed Google Scholar

El-Bassel N, Gilbert L, Wu E, Go H, Hill J. Relationship between drug abuse and intimate partner violence: a longitudinal study among women receiving methadone. Am J Public Health. 2005;95(3):465–70.

Hildebrand Karlén M, Roos af Hjelmsäter E, Fahlke C, Granhag PA, Söderpalm Gordh A. Alcohol intoxicated witnesses: perception of aggression and guilt in intimate Partner violence. J Interpers Violence. 2015;32(22):3448–74.

Kane JC, Van Wyk SS, Murray S, Bolton P, Melendez F, Danielson C, et al. Testing the effectiveness of a transdiagnostic treatment approach in reducing violence and alcohol abuse among families in Zambia: study protocol of the violence and Alcohol Treatment (VATU) trial. Global Mental Health. 2017;4:e18.

Download references

Acknowledgements

AcknowledgmentWe would like to thank the MEASUR DHS was approved to access this dataset to carry out this secondary data analysis.

Funding not applicable.

Author information

Authors and affiliations.

Department of Psychiatry, College of Medicine and Health Science, University of Gondar, Gondar, Ethiopia

Mamaru Melkam

Department of General Midwifery, School of Midwifery, College of Medicine & Health Sciences, University of Gondar, Gondar, Ethiopia

Bezawit Melak Fente

Department of Medicine, Adama General Hospital and Medical College, Adama University, Adama, Ethiopia

Yohannes Mekuria Negussie

Department of Ophthalmology, School of Medicine and Health Science, Debre Tabor University, Debre Tabor, Ethiopia

Zufan Alamrie Asmare

Department of Public Health, College of Medicine and Health Sciences, Samara University, Samara, Ethiopia

Hiwot Altaye Asebe & Beminate Lemma Seifu

Department of Public Health Officer, Institute of Public Health, College of Medicine and Health Sciences, University of Gondar, Gondar, Ethiopia

Meklit Melaku Bezie

Department of Epidemiology and Biostatistics, School of Public Health, College of Medicine and Health Sciences, Wolaita Sodo University, Wolaita Sodo, Ethiopia

Angwach Abrham Asnake

You can also search for this author in PubMed Google Scholar

Contributions

MM conceptualized the study and was involved in design, analysis, interpretation, and manuscript writing. AAA, BMF, YMN, ZAA, HAA, BLS, and MMB made a substantial contribution to the extraction of data, analysis, interpretation, drafting of the manuscript, and critical revision. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Mamaru Melkam .

Ethics declarations

Consent for publication.

Not applicable.

Competing interests

The authors declare no competing interests.

Ethics approval and consent to participate

Since we used secondary data and had no direct interaction with the study participants, ethical clearance was not required for this investigation. Study participants received written informed consent in return for their involvement. We have permission to access the data online by submitting a request to the DHS program’s measure at http://www.dhsprogram.com . The data was obtained via the program’s measure. The public can freely access information on the internet. The details of the ethical approval for the Demographic and Health Surveys (DHS) program make it possible to approve the download of survey data.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Melkam, M., Fente, B.M., Negussie, Y.M. et al. Impact of partner alcohol use on intimate partner violence among reproductive-age women in East Africa Demographic and Health Survey: propensity score matching. BMC Public Health 24 , 2365 (2024). https://doi.org/10.1186/s12889-024-19932-6

Download citation

Received : 14 June 2024

Accepted : 29 August 2024

Published : 30 August 2024

DOI : https://doi.org/10.1186/s12889-024-19932-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Intimate partner violence
Alcohol use
Propensity score matching
East Africa

BMC Public Health

ISSN: 1471-2458

General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
Explore content
About the journal
Publish with us
Sign up for alerts
28 August 2024
Update 02 September 2024

Exclusive: the papers that most heavily cite retracted studies

Richard Van Noorden &
Miryam Naddaf

You can also search for this author in PubMed Google Scholar

You have full access to this article via your institution.

Computer rendered illustration of a lone figure watching as a towering house of blank white cards collapses.

Credit: Waldemar Thaut/Zoonar via Alamy

In January, a review paper 1 about ways to detect human illnesses by examining the eye appeared in a conference proceedings published by the Institute of Electrical and Electronics Engineers (IEEE) in New York City. But neither its authors nor its editors noticed that 60% of the papers it cited had already been retracted.

The case is one of the most extreme spotted by a giant project to find papers whose results might be in question because they cite retracted or problematic research. The project’s creator, computer scientist Guillaume Cabanac at the University of Toulouse in France, shared his data with Nature ’s news team, which analysed them to find the papers that most heavily cite retracted work yet haven’t themselves been withdrawn (see ‘Retracted references’).

Chain retraction: how to stop bad science propagating through the literature

“We are not accusing anybody of doing something wrong. We are just observing that in some bibliographies, the references have been retracted or withdrawn, meaning that the paper may be unreliable,” Cabanac says. He calls his tool a Feet of Clay Detector, referring to an analogy, originally from the Bible, about statues or edifices that collapse because of their weak clay foundations.

The IEEE paper is the second-highest on the list assembled by Nature , with 18 of the 30 studies it cites withdrawn. Its authors didn’t respond to requests for comment, but IEEE integrity director Luigi Longobardi says that the publisher didn’t know about the issue until Nature asked, and that it is investigating.

Cabanac, a research-integrity sleuth, has already created software to flag thousands of problematic papers in the literature for issues such as computer-written text or disguised plagiarism . He hopes that his latest detector, which he has been developing over the past two years and describes this week in a Comment article in Nature , will provide another way to stop bad research propagating through the scientific literature — some of it fake work created by ‘papermill’ firms .

Further scrutiny

Cabanac lists the detector’s findings on his website , but elsewhere online — on the paper-review site PubPeer and on social media — he has explicitly flagged more than 1,700 papers that caught his eye because of their reliance on retracted work. Some authors have thanked Cabanac for alerting them to problems in their references. Others argue that it’s unfair to effectively cast aspersions on their work because of retractions made after publication that, they say, don’t affect their paper.

Scientific sleuths spot dishonest ChatGPT use in papers

Retracted references don’t definitively show that a paper is problematic, notes Tamara Welschot, part of the research-integrity team at Springer Nature in Dordrecht, the Netherlands, but they are a useful sign that a paper might benefit from further scrutiny. ( Nature ’s news team is independent of its publisher, Springer Nature.)

Some researchers argue that retraction of references in a narrative review — which describes the state of research in a field — doesn’t necessarily invalidate the original paper. But when studies assessed by a systematic review or meta-analysis are withdrawn, the results of that review should always be recalculated to keep the literature up to date, says epidemiologist Isabelle Boutron at Paris City University.

Retracted references

These studies have the highest proportion of retracted papers in their reference lists, according to Nature ’s analysis of articles flagged by the Feet of Clay Detector.

Year	Title of paper	Number of retracted studies in reference list
2012		33 of 51 (65%)
2023		18 of 30 (60%)
2024		46 of 77 (60%)
2012		25 of 53 (47%)
2001		25 of 53 (47%)
2016		15 of 33 (45%)
2012		40 of 125 (32%)
2013		18 of 57 (32%)
2012		47 of 225 (21%)
2023		12 of 58 (21%)

Source: Nature analysis of data from the Feet of Clay Detector . Figures for references and retractions were hand-checked and altered where necessary; detector data sources do not always give accurate counts.

Picking up fraudsters

Some of the papers that cite high proportions of retracted work are authored by known academic fraudsters who have had many of their own papers retracted.

These include engineering researcher Ali Nazari, who was dismissed from Swinburne University of Technology in Melbourne, Australia, in 2019, after a university misconduct investigation into his activities. He previously worked at Islamic Azad University in Saveh, Iran, and his current whereabouts are unclear. After Nature told publishers about his extant papers 2 , 3 topping Cabanac’s lists — including Elsevier and Fap-Unifesp, a non-profit foundation that supports the Federal University of São Paulo in Brazil — they said that they would look into the articles. One of the relevant journals was discontinued in 2013, Elsevier noted.

Cabanac’s detector also flags papers 4 by Chen-Yuan Chen, a computer scientist who worked at the National Pingtung University of Education in Taiwan until 2014. He was behind a syndicate that faked peer review and boosted citations, which came to light in 2014 after an investigation by the publisher SAGE. Some of Chen’s papers that are still in the literature were published by Springer Nature, which says it hadn’t been aware of the issue but is now investigating. Neither Chen nor Nazari responded to Nature ’s requests for comment.

Another flagged study 5 is by Ahmad Salar Elahi, a physicist affiliated with the Islamic Azad University in Tehran who has already had dozens of his papers retracted, in many cases because of excessive self-citation and instances of faked peer review. In 2018, the website Retraction Watch (which also wrote about the Nazari and Chen cases) reported that according to Mahmoud Ghoranneviss, then-director of the Plasma Physics Research Centre where Elahi worked, Elahi was likely to be dismissed from the university. Now, Ghoranneviss — who has retired — says that Elahi was barred only from that centre and not the rest of the university. Elahi continues to publish papers, sometimes listing co-authors including Ghoranneviss, who says he wasn’t aware of this. Neither Elahi nor the university responded to Nature ’s queries. The IEEE and Springer Nature, publishers of the journals that ran the Elahi papers, say they’re investigating.

Unhappy authors

Some authors are unhappy about Cabanac’s work. In May 2024, editors of the journal Clinical and Translational Oncology placed an expression of concern on a 2019 review paper 6 about RNA and childhood cancers, warning that it might not be reliable because it cited “a number of articles that have been retracted”. The journal’s publishing editor, Ying Jia at Springer Nature in Washington DC, says the team was alerted by one of Cabanac’s posts on social media last year.

Guillaume Cabanac poses for a portrait on the Paul Sabatier University campus.

Computer scientist Guillaume Cabanac has flagged more than 1,700 papers that caught his eye because of their reliance on retracted work. Credit: Fred Scheiber/SIPA/Shutterstock

Cabanac’s analysis finds that just under 10% of the article’s 637 references have been retracted — almost all after the review was published. However, the paper’s corresponding author, María Sol Brassesco, a biologist at the University of São Paulo, says that removing these references doesn’t change the conclusions of the review, and that she has sent the journal an updated version, which it hasn’t published. Because the cited works were retracted after publication, the expression of concern “felt like we were being punished for something that we could not see ahead”, she says. Jia says that editors felt that adding the notice was the most appropriate action.

In other cases, authors disagree about what to do. Nature examined three papers 7 , 8 , 9 in which between 5 and 16% of the references have now been retracted, all co-authored by Mohammad Taheri, a genetics PhD student at Friedrich Schiller University of Jena in Germany. He says that criticisms of his work on PubPeer “lack solid scientific basis”. Yet, in May, a co-author of two of those works, Marcel Dinger, dean of science at the University of Sydney in Australia, told the website For Better Science and Retraction Watch that he was reassessing review papers that cited retracted articles. He now says that his team has submitted corrections for the works, but Frontiers, which published one paper, says it hasn’t received the correspondence and will investigate. Elsevier — which published the other two papers — also says that it is examining the issue.

Catching problems early

Examples in which papers cite already-retracted work suggest that publishers could do a better job of screening manuscripts. For instance, 20 studies cited by a 2023 review paper 10 about RNA and gynaecological cancers in Frontiers in Oncology had been retracted before the article was submitted. Review co-author Maryam Mahjoubin-Tehran, a pharmacist at Mashhad University of Medical Sciences in Iran, told Nature that her team didn’t know about the retractions, and does not plan to update or withdraw the paper. The publisher, Frontiers, says it is investigating.

Until recently, publishers have not flagged citations to retracted papers in submitted manuscripts. However, many publishers say they are aware of Cabanac’s tool and monitor issues he raises, and some are bringing in similar screening tools.

Last year, Wiley announced it was checking Retraction Watch’s database of retracted articles to flag issues in reference lists, and Elsevier says it is also rolling out a tool that assesses manuscripts for red flags such as self-citations and references to retracted work. Springer Nature is piloting an in-house tool to look for retracted papers in manuscript citations and Longobardi says the IEEE is considering including Feet of Clay or similar solutions in its workflow. A working group for the STM Integrity Hub — a collaboration between publishers — has also tested the Feet of Clay Detector and “found it useful”, says Welschot.

Medical trend

Medical reviews that cite studies in areas later shown to be affected by fraud are a recurring theme in Cabanac’s findings.

In theory, meta-analyses or systematic reviews should be withdrawn or corrected if work they have cited goes on to be retracted, according to a policy issued in 2021 by the Cochrane Collaboration, an international group known for its gold-standard reviews of medical treatments .

Boutron, who directs Cochrane France in Paris, is using Cabanac’s tool to identify systematic reviews that cite retracted work, and to assess the impact the retracted studies had on the overall results.

However, a 2022 study 11 suggests that authors are often reluctant to update reviews, even when they are told the papers cite retracted work. Researchers e-mailed the authors of 88 systematic reviews that cited now-retracted studies in bone health by a Japanese fraudster, Yoshihiro Sato . Only 11 of the reviews were updated, the authors told Nature last year.

Retraction alerts

Authors aren’t routinely alerted if work cited in their past papers is withdrawn — although in recent years, paper-management tools for researchers such as Zotero and EndNote have incorporated Retraction Watch’s open database of retracted papers and have begun to flag papers that have been taken down. Cabanac thinks publishers might use tools like his to create similar alerts.

In 2016, researchers at the University of Oxford, UK, began developing a tool called RetractoBot , which automatically notifies authors by e-mail when a study that they have previously cited has been retracted. The software currently monitors 20,000 retracted papers and about 400,000 papers, published after 2000, that cite them. The team behind it is running a randomized trial to see whether papers flagged by RetractoBot are subsequently cited less than those not flagged by the tool, and will publish its results next year, says project lead Nicholas DeVito, a integrity researcher at Oxford.

The team has alerted more than 100,000 researchers so far. DeVito says that a minority of authors are annoyed about being contacted, but that others are grateful. “We are merely trying to provide a service to the community to reduce this practice from happening,” he says.

Nature 633 , 13-15 (2024)

doi: https://doi.org/10.1038/d41586-024-02719-5

Updates & Corrections

Update 02 September 2024 : This story has been updated to include mention of a website that reported Marcel Dinger’s comments relating to the citation of retracted papers.

Sandhiya, M. & Aneetha, A. S. 9th Intl Conf. Smart Struct. Syst . 1–4 (2023).

Nazari, A. Mater. Res. 15 , 383–396 (2012).

Article Google Scholar

Nazari, A., Khalaj, G. & Riahi, S. Math. Comput. Model. 55 , 1339–1353 (2012).

Shih, B.-Y., Chen, T.-H., Cheng, M.-H., Chen, C.-Y. & Chen, B.-W. Nat. Hazards 65 , 1637–1652 (2013).

Salar Elahi, A. & Ghoranneviss, M. IEEE Trans. Plasma Sci. 41 , 334–340 (2013).

Viera, G. M. et al. Clin. Transl. Oncol. 21 , 1583–1623 (2019); editorial expression of concern 26 , 1806 (2024).

Taheri, M. et al. Exp. Molec. Pathol . https://doi.org/10.1016/j.yexmp.2021.104602 (2021).

Taheri, M. et al. Front. Mol. Biosci . https://doi.org/10.3389/fmolb.2021.665199 (2021).

Ghafouri-Fard, S. et al. Biomed. Pharmacotherapy 137 , 111279 (2021).

Rezaee, A. et al. Front. Oncol. 13 , 1215194 (2023).

Article PubMed Google Scholar

Avenell, A., Bolland, M. J., Gamble, G. D. & Grey, A. Account. Res. 31 , 14–37 (2022).

Download references

Reprints and permissions

Supplementary Information

NEWS Retracted references Supplementary Information 2024

Peer review

Cash for errors: project offers bounty for spotting mistakes in published papers

Technology Feature 19 AUG 24

Who will make AlphaFold3 open source? Scientists race to crack AI model

News 23 MAY 24

Pay researchers to spot errors in published papers

World View 21 MAY 24

How can I publish open access when I can’t afford the fees?

Career Feature 02 SEP 24

Chain retraction: how to stop bad science propagating through the literature

Comment 28 AUG 24

No more hunting for replication studies: crowdsourced database makes them easy to find

Nature Index 27 AUG 24

Postdoc/PhD opportunity – Pharmacology of Opioids

Join us at MedUni Vienna to explore the pharmacology of circular and stapled peptide therapeutics targetting the κ-opioid receptor in the periphery.

Vienna (AT)

Medical University of Vienna

Division Director - Experimental Hematology and Cancer Biology

Cincinnati Children’s Hospital seeks the next Director for the Division of Experimental Hematology and Cancer Biology.

Cincinnati, Ohio

Cincinnati Children's Hospital & Medical Center

Faculty and Research Positions, Postdoctoral Recruitment

Jointly sponsored by the Hangzhou Municipal People's Government and the University of Chinese Academy of Sciences.

Hangzhou, Zhejiang, China

Hangzhou Institute of Advanced Study, UCAS

Associate or Senior Editor, Nature Energy

Job Title: Associate or Senior Editor, Nature Energy Location: New York, Jersey City, Philadelphia or London — Hybrid Working Application Deadline:...

New York City, New York (US)

Springer Nature Ltd

Faculty Positions & Postdocs at Institute of Physics (IOP), Chinese Academy of Sciences

IOP is the leading research institute in China in condensed matter physics and related fields. Through the steadfast efforts of generations of scie...

Beijing, China

Institute of Physics (IOP), Chinese Academy of Sciences (CAS)

Quick links

Explore articles by subject
Guide to authors
Editorial policies

IMAGES

How to Write a Stellar Literature Review
50 Smart Literature Review Templates (APA) ᐅ TemplateLab
How to write a literature review in research paper
Literature Review Table Template
The Literature Review Definition A literature review surveys
(DOC) An Overview of Literature Survey Review: The Foundation of

VIDEO

Gujarati Literature Part 3 of 4
Introduction to Literature Review, Systematic Review, and Meta-analysis
Review of literature|| Review of literature
Research Methods: Lecture 3
What is Literature Review?| How to write Literature review?| Research Methodology|
Literature Survey

COMMENTS

How to Write a Literature Review
A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question. It is often written as part of a thesis, dissertation, or research paper, in order to situate your work in relation to existing knowledge.
What is a Literature Review? How to Write It (with Examples)
A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing ...
5. The Literature Review
A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated.
Writing a Literature Review
Writing a Literature Review. A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and ...
How To Write A Literature Review
"A literature review surveys books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated.
What is a Literature Review?
A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research. There are five key steps to writing a literature review: Search for relevant literature. Evaluate sources. Identify themes, debates and gaps.
Literature Review: Definition and Context
Literature Review is one part of that process of writing a research paper. In a research paper, you use the literature as a starting point, a building block and as evidence of a new insight. The goal of the literature review is only to summarize and synthesize the arguments and ideas of others. You should not present your original idea.
Literature Review: The What, Why and How-to Guide
Example: Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework: 10.1177/08948453211037398 ; Systematic review: "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139).
What is a Literature Review?
A literature review surveys scholarly articles, books and other sources relevant to a particular issue, area of research, or theory. The purpose is to offer an overview of significant literature published on a topic. A literature review may constitute an essential chapter of a thesis or dissertation, or may be a self-contained review of writings on a subject.
PDF Writing an Effective Literature Review
A literature review is a survey of published work relevant to a particular issue, field of research, topic or theory. It will never be about everything and should have clearly defined limits. This survey will certainly provide short descriptions of the sources being reviewed, but much more importantly it will also provide the reader with a
Steps in Conducting a Literature Review
A literature review is an integrated analysis-- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.
Ten Simple Rules for Writing a Literature Review
Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications .For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively .Given such mountains of papers, scientists cannot be expected to examine in detail every ...
Literature Review Research
Literature Review is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.. Also, we can define a literature review as the collected body of scholarly works related to a topic:
Literature review as a research methodology: An ...
As mentioned previously, there are a number of existing guidelines for literature reviews. Depending on the methodology needed to achieve the purpose of the review, all types can be helpful and appropriate to reach a specific goal (for examples, please see Table 1).These approaches can be qualitative, quantitative, or have a mixed design depending on the phase of the review.
What Is A Literature Review?
The word "literature review" can refer to two related things that are part of the broader literature review process. The first is the task of reviewing the literature - i.e. sourcing and reading through the existing research relating to your research topic. The second is the actual chapter that you write up in your dissertation, thesis or ...
How To Write A Literature Review (+ Free Template)
Okay - with the why out the way, let's move on to the how. As mentioned above, writing your literature review is a process, which I'll break down into three steps: Finding the most suitable literature. Understanding, distilling and organising the literature. Planning and writing up your literature review chapter.
Steps in the Literature Review Process
Literature Review and Research Design by Dave Harris This book looks at literature review in the process of research design, and how to develop a research practice that will build skills in reading and writing about research literature--skills that remain valuable in both academic and professional careers. Literature review is approached as a process of engaging with the discourse of scholarly ...
LSBU Library: Literature Reviews: What is a Literature Review?
A literature review is an academic text that surveys, synthesizes, and critically evaluates the existing literature on a specific topic. It is typically required for theses, dissertations, or long reports and serves several key purposes:
research process
The results of the literature survey can contribute to the body of knowledge when peer-reviewed and published as survey articles. Literature Review: Is the process of technically and critically reviewing published papers to extract technical and scientific metadata from the presented contents. The metadata are usually used during literature ...
Preparing Research Reports and Integrative Literature Reviews
Literature Review: This section situates this study within the larger body of literature. Although professional communication is an interdisciplinary field and readers have eclectic interests, the one thing that connects readers of this journal is their interest in professional communication. ... For instruments, like surveys, provide a summary ...
Literature Review Guidelines
Your literature review must include enough works to provide evidence of both the breadth and the depth of the research on your topic or, at least, one important angle of it. The number of works necessary to do this will depend on your topic. For most topics, AT LEAST TEN works (mostly books but also significant scholarly articles) are necessary ...
"Evaluating The Impact of The Nurse-Patient Relationship: An Integrativ
The literature review explored the importance of fostering the nurse-patient relationship and its impact on patient satisfaction and quality. Utilizing the PRISMA model and Melnyk's level of evidence, 25 articles were chosen for the literature review. ... The star ratings on the surveys have a direct impact on the home care agency and all of ...
Presentation Attack Detection: A Systematic Literature Review
A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness. Next. Abstract; References; ... evaluation methods, and attack types. In this systematic literature review, we identify and categorise the state-of-the-art approaches in each domain to cover the challenges and solutions in a single ...
Advances, recognition, and interpretation of molecular heterogeneity
Learn more about Advances, recognition, and interpretation of molecular heterogeneity among conventional and subtype histology of urothelial carcinoma (UC): a survey among urologic pathologists and comprehensive review of the literature. 07/29/2024
Software product line testing: a systematic literature review
This paper aims to survey existing research on SPL testing to provide researchers and practitioners with up-to-date evidence and issues that enable further development of the field. To this end, we conducted a Systematic Literature Review (SLR) with seven research questions in which we identified and analyzed 118 studies dating from 2003 to 2022.
A scoping review of stroke services within the Philippines
This scoping review aimed to map available literature on stroke services in the Philippines, based on Arksey and O'Malley's five-stage-process. ... (n = 31), two were audits, eight were surveys or questionnaires, four were consensus opinion and/or guideline development, three were randomized controlled trial (RCT) or feasibility RCT, 1 was ...
Chinese College Student Financial Literacy: Knowledge, Attitude, and
The "campus loans" crisis has highlighted the importance of financial literacy among Chinese college students. Based on an analysis of 2,266 valid questionnaires, this study utilized survey data and logistic regression to examine the correlations between demographic and behavioral factors and financial literacy among students.
Impact of partner alcohol use on intimate partner violence among
Intimate Partner Violence (IPV) is the most prevalent form of violence against women globally and is more prevalent than rape or other violent attacks by strangers. Different observational studies have established a strong positive association between alcohol use and intimate partner violence. Even though there are a lot of studies that show the association between partner alcohol use and ...
Exclusive: the papers that most heavily cite retracted studies
Credit: Waldemar Thaut/Zoonar via Alamy. In January, a review paper 1 about ways to detect human illnesses by examining the eye appeared in a conference proceedings published by the Institute of ...

Have a language expert improve your writing

How to Write a Literature Review | Guide, Examples, & Templates

Instantly correct all language mistakes in your text

Table of contents

Receive feedback on language, structure, and formatting

Make a list of keywords

Search for relevant sources

Take notes and cite your sources

Don't submit your assignments before you do this

Chronological

Methodological

Theoretical

Cite this Scribbr article

Is this article helpful?

Shona McCombes

What is a Literature Review? How to Write It (with Examples)

Table of Contents

Frequently asked questions

Literature review example

Literature Review on Climate Change Impacts on Biodiversity:

How to write a good literature review

Conducting a literature review

Choose a Topic and Define the Research Question:

Decide on the Scope of Your Review:

Select Databases for Searches:

Conduct Searches and Keep Track:

Review the Literature:

Organize and Write Your Literature Review:

How to write a literature review faster with Paperpal?

Related Reads:

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Organizing Your Social Sciences Research Paper

Importance of a Good Literature Review

Structure and Writing Style

Writing Tip

Another Writing Tip

Yet Another Writing Tip

Writing a Literature Review

Welcome to the Purdue OWL

Where, when, and why would I write a lit review?

What are the parts of a lit review?

How should I organize my lit review?

What are some strategies or tips I can use while writing my lit review?

Have a language expert improve your writing

What is a Literature Review? | Guide, Template, & Examples

Instantly correct all language mistakes in your text

Table of contents

Prevent plagiarism, run a free check.

Make a list of keywords

Search for relevant sources

Take notes and cite your sources

Chronological

Methodological

Theoretical

Cite this Scribbr article

Is this article helpful?

Shona McCombes

Survey Research: Design and Presentation

Literature Review for Grad Students in Education

Introduction to Literature Review

What is a Literature Review

Purpose of Literature Review?

Structure of Literature Review

Different Types of Literature Sources

Literature Review: The What, Why and How-to Guide — Introduction

What are Literature Reviews?

Goals of Literature Reviews

Types of Literature Reviews

Literature Reviews in the Health Sciences

Writing Seminar 163/164 And the Rest is Drag: What is a Literature Review?

Finding Examples

What is a literature review?

Harvey Cushing/John Hay Whitney Medical Library

YSN Doctoral Programs: Steps in Conducting a Literature Review

What is a literature review?

APA7 Style resources

1. Choose a topic. Define your research question.

2. Decide on the scope of your review

3. Select the databases you will use to conduct your searches.

4. Conduct your searches to find the evidence. Keep track of your searches.