kelemahan systematic literature review

Strengths and Weaknesses of Systematic Reviews

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

Systematic reviews are considered credible sources since they are comprehensive, reproducible, and precise in stating the outcomes. The type of review system used and the approach taken depend on the goals and objectives of the research. To choose the best-suited review system, researchers must be aware of the strengths and weaknesses of each one.

Let us now look at the strengths and limitations of systematic reviews.

Strengths Of Systematic Reviews

Systematic reviews have become increasingly popular owing to their transparency, accuracy, replicability, and reduced risk of bias. Some of the main benefits of systematic reviews are;

Specificity

Researchers can answer specific research questions of high importance. For example, the efficacy of a particular drug in the treatment of an illness.

Explicit Methodology

A systematic review requires rigorous planning. Each stage of the review is predefined to the last detail. The research question is formulated using the PICO (population, intervention, comparison, and outcome) approach. A strict eligibility criteria is then established for inclusion and exclusion criteria for selecting the primary studies for the review. Every stage of the systematic review methodology is pre-specified to the last detail and made publicly available, even before starting the review process. This makes all the stages in the methodology transparent and reproducible.

Reliable And Accurate Results

The results of a systematic review are either analyzed qualitatively and presented as a textual narrative or quantitatively using statistical methods such as meta-analyses and numeric effect estimates. The quality of evidence or the confidence in effect estimates is calculated using the standardized GRADE approach.

Comprehensive And Exhaustive

A systematic review involves a thorough search of all the available data on a certain topic. It is exhaustive and considers every bit of evidence in synthesizing the outcome. Primary sources for the review are collected from databases and multiple sources, such as blogs from pharmaceutical companies, unpublished research directly from researchers, government reports, and conference proceedings. These are referred to as grey literature. The search criteria and keywords used in sourcing are specific and predefined.

Reproducible

Learn more about distillersr.

(Article continues below)

Weaknesses Of Systematic Reviews

Although systematic reviews are robust tools in scientific research they are not immune to errors. They can be misleading, or even harmful if the data is inappropriately handled or if they are biased. Some of the limitations of systematic reviews include:

Mass Production

Due to the popularity systematic reviews have gained, they tend to be used more than required. The growth rate of systematic reviews has outpaced the growth rate of studies overall. This results in redundancy. For example, a survey published in the BMJ[1], included 73 randomly selected meta-analyses published in 2010 found that for two-thirds of these studies, there was at least one, and sometimes as many as 13, additional meta-analyses published on the same topic by early 2013.

Risk of Bias

Although systematic reviews have many advantages, they are also more susceptible to certain types of biases. A bias is a systematic or methodological error that causes misrepresentation of the study outcomes. As bias can appear at any stage, authors should be aware of the specific risks at each stage of the review process. Most of the known errors in systematic reviews arise in the selection and publication stages. The eligibility criterion in a systematic review helps to avoid selection bias. Poor study design and execution can also result in a biased outcome. It’s important to learn about the types of bias in systematic reviews .

Expressing Strong Opinions by Stealth

Selective outcome reporting is a major threat to a systematic review. The author or reviewer may decide to only report a selection of the statistically significant outcomes that suit his interest. The possibility of unfair or misleading interpretation of evidence outcomes in a systematic review can have serious implications.

Like any review system, systematic reviews have their advantages and disadvantages. Understanding them is essential to making a choice of which review system to use.

Overlapping meta-analyses on the same topic: survey of published studies. BMJ 2013; 347:f4501

3 Reasons to Connect

Eight problems with literature reviews and how to fix them

Affiliations.

• 1 Mercator Research Institute on Climate Change and Global Commons, Berlin, Germany. [email protected].
• 2 Stockholm Environment Institute, Stockholm, Sweden. [email protected].
• 3 Africa Centre for Evidence, University of Johannesburg, Johannesburg, South Africa. [email protected].
• 4 College of Medicine and Health, Exeter University, Exeter, UK.
• 5 Department of Zoology, University of Cambridge, Cambridge, UK.
• 6 School of Biological Sciences, University of East Anglia, Norwich, UK.
• 7 Department of Biological Sciences, Royal Holloway University of London, Egham, UK.
• 8 Stockholm Environment Institute, Stockholm, Sweden.
• 9 Department of Zoology, University of Oxford, Oxford, UK.
• 10 Collaboration for Environmental Evidence, UK Centre, School of Natural Sciences, Bangor University, Bangor, UK.
• 11 Liljus ltd, London, UK.
• 12 Department of Forest Sciences, University of Helsinki, Helsinki, Finland.
• 13 Evidence Synthesis Lab, School of Natural and Environmental Sciences, University of Newcastle, Newcastle-upon-Tyne, UK.
• PMID: 33046871
• DOI: 10.1038/s41559-020-01295-x

Traditional approaches to reviewing literature may be susceptible to bias and result in incorrect decisions. This is of particular concern when reviews address policy- and practice-relevant questions. Systematic reviews have been introduced as a more rigorous approach to synthesizing evidence across studies; they rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing susceptibility to bias. Despite the increasing popularity of systematic reviews in the environmental field, evidence synthesis methods continue to be poorly applied in practice, resulting in the publication of syntheses that are highly susceptible to bias. Recognizing the constraints that researchers can sometimes feel when attempting to plan, conduct and publish rigorous and comprehensive evidence syntheses, we aim here to identify major pitfalls in the conduct and reporting of systematic reviews, making use of recent examples from across the field. Adopting a 'critical friend' role in supporting would-be systematic reviews and avoiding individual responses to police use of the 'systematic review' label, we go on to identify methodological solutions to mitigate these pitfalls. We then highlight existing support available to avoid these issues and call on the entire community, including systematic review specialists, to work towards better evidence syntheses for better evidence and better decisions.

• Environment
• Research Design
• Systematic Reviews as Topic*

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.8(11); 2016 Nov

How to Conduct a Systematic Review: A Narrative Literature Review

Nusrat jahan.

1 Psychiatry, Mount Sinai Chicago

Sadiq Naveed

2 Psychiatry, KVC Prairie Ridge Hospital

Muhammad Zeshan

3 Department of Psychiatry, Bronx Lebanon Hospital Icahn School of Medicine at Mount Sinai, Bronx, NY

Muhammad A Tahir

4 Psychiatry, Suny Upstate Medical University, Syracuse, NY

Systematic reviews are ranked very high in research and are considered the most valid form of medical evidence. They provide a complete summary of the current literature relevant to a research question and can be of immense use to medical professionals. Our goal with this paper is to conduct a narrative review of the literature about systematic reviews and outline the essential elements of a systematic review along with the limitations of such a review.

Introduction and background

A literature review provides an important insight into a particular scholarly topic. It compiles published research on a topic, surveys different sources of research, and critically examines these sources [ 1 ]. A literature review may be argumentative, integrative, historical, methodological, systematic, or theoretical, and these approaches may be adopted depending upon the types of analysis in a particular study [ 2 ].

Our topic of interest in this article is to understand the different steps of conducting a systematic review. Systematic reviews, according to Wright, et al., are defined as a “review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review” [ 3 ]. A systematic review provides an unbiased assessment of these studies [ 4 ]. Such reviews emerged in the 1970s in the field of social sciences. Systematic reviews, as well as the meta-analyses of the appropriate studies, can be the best form of evidence available to clinicians [ 3 ]. The unsystematic narrative review is more likely to include only research selected by the authors, which introduces bias and, therefore, frequently lags behind and contradicts the available evidence [ 5 ].

Epidemiologist Archie Cochrane played a vital role in formulating the methodology of the systematic review [ 6 ]. Dr. Cochrane loved to study patterns of disease and how these related to the environment. In the early 1970s, he found that many decisions in health care were made without reliable, up-to-date evidence about the treatments used [ 6 ].

A systematic review may or may not include meta-analysis, depending on whether results from different studies can be combined to provide a meaningful conclusion. David Sackett defined meta-analysis as a “specific statistical strategy for assembling the results of several studies into a single estimate” [ 7 - 8 ].

While the systematic review has several advantages, it has several limitations which can affect the conclusion. Inadequate literature searches and heterogeneous studies can lead to false conclusions. Similarly, the quality of assessment is an important step in systematic reviews, and it can lead to adverse consequences if not done properly.

The purpose of this article is to understand the important steps involved in conducting a systematic review of all kinds of clinical studies. We conducted a narrative review of the literature about systematic reviews with a special focus on articles that discuss conducting reviews of randomized controlled trials. We discuss key guidelines and important terminologies and present the advantages and limitations of systematic reviews.

Narrative reviews are a discussion of important topics on a theoretical point of view, and they are considered an important educational tool in continuing medical education [ 9 ]. Narrative reviews take a less formal approach than systematic reviews in that narrative reviews do not require the presentation of the more rigorous aspects characteristic of a systematic review such as reporting methodology, search terms, databases used, and inclusion and exclusion criteria [ 9 ]. With this in mind, our narrative review will give a detailed explanation of the important steps of a systematic review.

Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) checklist

Systematic reviews are conducted based on predefined criteria and protocol. The PRISMA-P checklist, developed by Moher, et al., contains 17 items (26 including sub-items) comprising the important steps of a systematic review, including information about authors, co-authors, their mailing and email addresses, affiliations, and any new or updated version of a previous systematic review [ 9 ]. It also identifies a plan for documenting important protocol amendments, registry names, registration numbers, financial disclosures, and other support services [ 10 ]. Moher, et al. also state that methods of systematic reviews involve developing eligibility criteria and describing information sources, search strategies, study selection processes, outcomes, assessment of bias in individual studies, and data synthesis [ 10 ].

Research question

Writing a research question is the first step in conducting a systematic review and is of paramount importance as it outlines both the need and validity of systematic reviews (Nguyen, et al., unpublished data). It also increases the efficiency of the review by limiting the time and cost of identifying and obtaining relevant literature [ 11 ]. The research question should summarize the main objective of a systematic review.

An example research question might read, “How does attention-deficit/hyperactivity disorder (ADHD) affect the academic performance of middle school children in North America?” The question focuses on the type of data, analysis, and topic to be discussed (i.e., ADHD among North American middle school students). Try to avoid research questions that are too narrow or broad—they can lead to the selection of only a few studies and the ability to generalize results to any other populations may be limited. An example of a research question that is too narrow would be, “What is the prevalence of ADHD in children and adolescents in Chicago, IL?” Alternately, if the research question is too broad, it can be difficult to reach a conclusion due to poor methodology. An example of a research question that is too broad in scope would be, “What are the effects of ADHD on the functioning of children and adolescents in North America?”

Different tools that can be used to help devise a research question, depending on the type of question, are: population, intervention, comparator, and outcomes (PICO); sample, phenomenon of interest, design, evaluation, and research type (SPIDER); setting, perspective, intervention, comparison, and evaluation (SPICE); and expectation, client group, location, impact, professionals, and service (ECLIPSE).

The PICO approach is mostly used to compare different interventions with each other. It helps to formulate a research question related to prognosis, diagnosis, and therapies [ 12 ].

Scenario: A 50-year-old white woman visited her psychiatrist with a diagnosis of major depressive disorder. She was prescribed fluoxetine, which she feels has been helpful. However, she experienced some unpleasant side effects of nausea and abdominal discomfort. She has recently been told by a friend about the use of St. John’s wort in treating depression and would like to try this in treating her current depression. (Formulating research questions, unpublished data).

In the above-mentioned scenario, the sample population is a 50-year-old female with major depressive disorder; the intervention is St. John’s wort; the comparison is fluoxetine; and the outcome would be efficacy and safety. In order to see the outcome of both efficacy and safety, we will compare the efficacy and safety of both St. John’s wort and fluoxetine in a sample population for treating depression. This scenario represents an example where we can apply the PICO approach to compare two interventions.

In contrast, the SPIDER approach is focused more on study design and samples rather than populations [ 13 ]. The SPIDER approach can be used in this research question: “What is the experience of psychiatry residents attending a transgender education?” The sample is psychiatry residents; the phenomenon of interest is transgender education; the design is a survey; the evaluation looks at the experience; and the research type is qualitative. 

The SPICE approach can be used to evaluate the outcome of a service, intervention, or project [ 14 ]. The SPICE approach applies to the following research question: “In psychiatry clinics, does the combined use of selective serotonin reuptake inhibitor (SSRI) and psychotherapy reduce depression in an outpatient clinic versus SSRI therapy alone?” The setting is the psychiatry clinic; the perspective/population is the outpatient; the intervention is combined psychotherapy and SSRI; the comparison is SSRI alone; and the evaluation is reduced depression. 

The ECLIPSE approach is useful for evaluating the outcome of a policy or service (Nguyen, et al., unpublished data). ECLIPSE can apply in the following research question: “How can a resident get access to medical records of patients admitted to inpatient from other hospitals?” The expectation is: “What are you looking to improve/change to increase access to medical records for patients admitted to inpatient?” The client group is the residents; the location is the inpatient setting; the impact would be the residents having easy access to medical records from other hospitals; and the professionals in this scenario would be those involved in improving the service experiences such as hospital administrators and IT staff.

Inclusion and exclusion criteria

Establishing inclusion and exclusion criteria come after formulating research questions. The concept of inclusion and exclusion of data in a systematic review provides a basis on which the reviewer draws valid and reliable conclusions regarding the effect of the intervention for the disorder under consideration [ 11 ]. Inclusions and exclusion are based on preset criteria for specific systematic review. It should be done before starting the literature search in order to minimize the possibility of bias.

Eligibility criteria provide the boundaries of the systematic review [ 15 ]. Participants, interventions, and comparison of a research question provide the basis for eligibility criteria [ 15 ]. The inclusion criteria should be able to identify the studies of interest and, if the inclusion criteria are too broad or too narrow, it can lead to an ineffective screening process.

Protocol registration

Developing and registering research protocol is another important step of conducting a systematic review. The research protocol ensures that a systematic review is carefully planned and explicitly documented before the review starts, thus promoting consistency in conduct for the review team and supporting the accountability, research integrity, and transparency of the eventually completed review [ 10 ]. PROSPERO and the Cochrane Database of Systematic Reviews are utilized for registering research protocols and research questions, and they check for prior existing duplicate protocols or research questions. PROSPERO is an international database of prospectively registered systematic reviews related to health care and social sciences (PRISMA, 2016). It is funded by the National Institute for Health Research. The Cochrane Collaboration concentrates on producing systematic reviews of interventions and diagnostic test accuracy but does not currently produce reviews on questions of prognosis or etiology [ 16 ].

A detailed and extensive search strategy is important for the systematic review since it minimizes bias in the review process [ 17 ].

Selecting and searching appropriate electronic databases is determined by the topic of interest. Important databases are: MEDLARS Online (MEDLINE), which is the online counterpart to the Medical Literature Analysis and Retrieval System (MEDLARS); Excerpta Medica Database (EMBASE); and Google Scholar. There are multiple electronic databases available based on the area of interest. Other important databases include: PsycINFO for psychology and psychiatry; Allied and Complementary Medicine Database (AMED) for complementary medicine; Manual, Alternative, and Natural Therapy Index System (MANTIS) for alternative medical literature; and Cumulative Index to Nursing and Allied Health Literature (CINAHL) for nursing and allied health [ 15 ].

Additional studies relevant for the review may be found by looking at the references of studies identified by different databases [ 15 ]. Non-indexed articles may be found by searching the content of journals, conferences proceedings, and abstracts. It will also help with letters and commentaries which may not get indexed [ 15 ]. Reviewing clinical trial registries can provide information about any ongoing trials or unpublished research [ 15 ]. A gray literature search can access unpublished papers, reports, and conference reports, and it generally covers studies that are published in an informal fashion, rather than in an indexed journal [ 15 ]. Further search can be performed by selecting important key articles and going through in-text citations [ 15 ].

Using Boolean operators, truncation, and wildcards

Boolean operators use the relationship between different search words to help with the search strategy. These are simple words (i.e., AND, OR, and NOT) which can help with more focused and productive results (poster, Jahan, et al.: How to conduct a systematic review. APPNA 39th Summer Convention. Washington, DC. 2016). The Boolean operator AND finds articles with all the search words. The use of OR broadens the focus of the search, and it will include articles with at least one search term. The researchers can also ignore certain results from the records by using NOT in the search strategy.

An example of AND would be using “depression” AND “children” in the search strategy with the goal of studying depression in children. This search strategy will include all the articles about both depression and children. The researchers may use OR if the emphasis of the study is mood disorders or affective disorders in adolescents. In that case, the search strategy will be “mood disorders” OR “affective disorders” AND “adolescents.” This search will find all the articles about mood disorders or affective disorders in adolescents. The researchers can use NOT if they only want to study depression in children and want to ignore bipolar disorder from the search. An example search in this scenario would be “depression” NOT “bipolar disorder” AND “children.” This will help ignore studies related to bipolar disorder in children.

Truncation and wildcards are other tools to make search strategy more comprehensive and focused. While the researchers search a database for certain articles, they frequently face terminologies that have the same initial root of a word but different endings. An example would be "autism," "autistic," and "autism spectrum disorder." These words have a similar initial root derived from “autis” but they end differently in each case. The truncation symbol (*) retrieves articles that contain words beginning with “autis” plus any additional characters. Wildcards are used for words with the same meanings but different spellings due to various reasons. For the words with spelling variations of a single letter, wildcard symbols can be used. When the researcher inputs “M+N” in the search bar, this returns results containing both “man” or “men” as the wildcard accounts for the spelling variations between the letters M and N.

Study selection

Study selection should be performed in a systematic manner, so reviewers deal with fewer errors and a lower risk of bias (online course, Li T, Dickersin K: Introduction to systematic review and meta-analysis. 2016. https://www.coursera.org/learn/systematic-review #). Study selection should involve two independent reviewers who select studies using inclusion and exclusion criteria. Any disagreements during this process should be resolved by discussion or by a third reviewer [ 10 ]. Specific study types can be selected depending on the research question. For example, questions on incidence and prevalence can be answered by surveys and cohort studies. Clinical trials can provide answers to questions related to therapy and screening. Queries regarding diagnostic accuracy can be answered by clinical trials and cross-sectional studies (online course, Li T, Dickersin K: Introduction to systematic review and meta-analysis. 2016. https://www.coursera.org/learn/systematic-review #). Prognosis and harm-related questions should use cohort studies and clinical trials, and etiology questions should use case-control and cohort studies (online course, Li T, Dickersin K: Introduction to systematic review and meta-analysis. 2016. https://www.coursera.org/learn/systematic-review #).

Data screening and data extractions are two of the major steps in conducting a systematic review [ 18 ]. Data screening involves searching for relevant articles in different databases using keywords. The next step of data screening is manuscript selection by reviewing each manuscript in the search results to compare that manuscript against the inclusion criteria [ 18 ]. The researchers should also review the references of the papers selected before selecting the final paper, which is the last step of data screening [ 18 ].

The next stage is extracting and appraising the data of the included articles [ 18 ]. A data extraction form should be used to help reduce the number of errors, and more than one person should record the data [ 17 ]. Data should be collected on specific points like population type, study authors, agency, study design, humanitarian crisis, target age groups, research strengths from the literature, setting, study country, type(s) of public health intervention, and health outcome(s) addressed by the public health intervention. All this information should then be put into an electronic database [ 18 ].

Assessing bias

Bias is a systematic error (or deviation from the truth) in results or inferences. Biases can change the results of any study and lead to an underestimation or overestimation of the true intervention effect [ 19 ]. Biases can impact any aspect of a review, including selecting studies, collecting and extracting data, and making a conclusion. Biases can vary in magnitude; some are small, with negligible effect, but some are substantial to a degree where an apparent finding may be entirely due to bias [ 19 ]. There are different types of bias, including, but not limited to, selection, detection, attrition, reporting, and performance.

Selection bias occurs when a sample selected is not representative of the whole general population. If randomization of the sample is done correctly, then chances of selection bias can be minimized [ 20 ].

Detection bias refers to systematic differences between groups in how outcomes are determined. This type of bias is based on knowledge of the intervention provided and its outcome [ 19 ].

Attrition bias refers to systematic differences between groups in withdrawals from a study [ 19 ]. The data will be considered incomplete if some subjects are withdrawn or have irregular visits during data collection.

Reporting bias refers to systematic differences between reported and unreported findings, and it is commonly seen during article reviews. Reporting bias is based on reviewer judgment about the outcome of selected articles [ 20 ].

Performance bias develops due to the knowledge of the allocated interventions by participants and personnel during the study [ 20 ]. Using a double-blind study design helps prevent performance bias, where neither the experimenter nor the subjects know which group contains controls and which group contains the test article [ 14 ].

Last step of systematic review: discussion

The discussion of a systematic review is where a summary of the available evidence for different outcomes is written and discussed [ 10 ]. The limitations of a systematic review are also discussed in detail. Finally, a conclusion is drawn after evaluating the results and considering limitations [ 10 ].

Discussion of the current article

Systematic reviews with or without a meta-analysis are currently ranked to be the best available evidence in the hierarchy of evidence-based practice [ 21 ]. We have discussed the methodology of a systematic review. A systematic review is classified in the category of filtered information because it appraises the quality of the study and its application in the field of medicine [ 21 ]. However, there are some limitations of the systematic review, as we mentioned earlier in our article. A large randomized controlled trial may provide a better conclusion than a systematic review of many smaller trials due to their larger sample sizes [ 22 ], which help the researchers generalize their conclusions for a bigger population. Other important factors to consider include higher dropout rates in large studies, co-interventions, and heterogeneity among studies included in the review.

As we discussed the limitations of the systematic review and its effect on quality of evidence, there are several tools to rate the evidence, such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system [ 22 ]. GRADE provides a structured approach to evaluating the risk of bias, serious inconsistency between studies, indirectness, imprecision of the results, and publication bias [ 22 ]. Another approach used to rate the quality of evidence is a measurement tool to assess systematic reviews (AMSTAR) [ 23 ]. It is also available in several languages [ 23 ].

Conclusions

Despite its limitations, a systematic review can add to the knowledge of the scientific community especially when there are gaps in the existing knowledge. However, conducting a systematic review requires different steps that involve different tools and strategies. It can be difficult at times to access and utilize these resources. A researcher can understand and strategize a systematic review following the different steps outlined in this literature review. However, conducting a systematic review requires a thorough understanding of all the concepts and tools involved, which is an extensive endeavor to be summed up in one article.

The Cochrane Handbook for Systematic Reviews of Interventions and the Center for Reviews and Dissemination (CRD) provide excellent guidance through their insightful and detailed guidelines. We recommend consulting these resources for further guidance.

Given that our article is a narrative review of the scholarly literature, it contains the same limitations as noted for any narrative review. We hope that our review of the means and methods for conducting a systematic review will be helpful in providing basic knowledge to utilize the resources available to the scientific community.

The content published in Cureus is the result of clinical experience and/or research by independent individuals or organizations. Cureus is not responsible for the scientific accuracy or reliability of data or conclusions published herein. All content published within Cureus is intended only for educational, research and reference purposes. Additionally, articles published within Cureus should not be deemed a suitable substitute for the advice of a qualified health care professional. Do not disregard or avoid professional medical advice due to content published within Cureus.

The authors have declared that no competing interests exist.

The future of evidence synthesis (2 of 3): The strengths and weaknesses of systematic reviews

Written By:

Maria Dellapina

Welcome to post two of a three-part Prism Academy series on systematic reviews! If you are thinking to yourself, “wait...post two?” be sure to check out the first post here . Trust me, it will help!

For those of you who don’t already know, my name is Maria, and I am the Head of Operations at Prism. Prior to joining the team, I was working in academia, coordinating research projects and spending a lot of time thinking about and synthesizing evidence. I want to thank you for joining me on this three-part series, where we are exploring the basics of systematic reviews, their current state, and how we must adapt them to keep pace with the world today. In today’s post, we are going to explore the strengths and weaknesses of systematic reviews and why bringing them up to speed with the pace of modern research is crucial to the everlasting pursuit of evidence-based practice.

But before we get too into the weeds, let's quickly discuss a bit of history:

Systematic reviews, as we know them today, were popularized in the 1970s when researchers at Oxford began synthesizing evidence in the pursuit of summarizing the effectiveness of health care interventions. This work, laid the foundation for Cochrane, "a global independent network of researchers, professionals, patients, carers and people interested in health." Given their rigorous methodology, Cochrane reviews are considered among the highest quality systematic reviews, informing evidence-based practice for researchers and practitioners around the globe.

In recent years, systematic reviews have rapidly increased in popularity, with some estimates claiming that in 2019, over 80 systematic reviews were published every day . That represents a ton of work - and we’ll come back to discuss more about that in a moment.

As previously mentioned, systematic reviews are considered by many to be the gold standard of evidence synthesis. There are several good reasons for this:

• Systematic reviews utilize rigorous methods which aim to minimize bias in the review of evidence from individual studies.
• Systematic reviews attempt to be comprehensive in their search strategy, enabling reviewers to look at all of the available evidence and combine it accordingly.
• Systematic reviews, when paired with an appropriately conducted meta-analysis, can allow reviewers to better understand the efficacy and effect size of a given intervention, analyze safety risks and benefits more comprehensively, extrapolate findings to the larger population, and examine sub-populations.
• Systematic reviews tend to include only trials of high-quality, such as randomized controlled trials (RCTs), however, systematic reviews are conducted on other types of studies.
• Systematic reviews are often conducted using standardized quality-assurance practices and guidelines such as PRISMA, the Cochrane Handbook for Systematic Reviews, and GRADE.

Given all of the above, it is easy to see why systematic reviews have been relied upon for decades as a source of evidence-based answers to questions that critically affect the practice of researchers, clinicians, or practitioners.

Yet, systematic reviews are far from a perfect solution to evidence synthesis. Indeed, their drawbacks quickly become apparent when one examines them within the context of today’s biomedical research ecosystem. Some of the weaknesses include:

• Systematic reviews usually take several months to complete, and by the time the results are published, the findings may be out of date. Even if attempts to update the review are made, these too may take many months.
• As I already mentioned above, some researchers estimate that over 80 systematic reviews are published every day . Given that many reviews are 20-80+ pages in length, if one new review is published in a particular field each week, the consumers of this knowledge (i.e., busy academics and clinicians) will likely struggle to keep up.
• Given the sheer quantity of systematic reviews being published, there is a substantial amount of unnecessary duplication .
• Despite the presence of standardized guidelines and practices, many systematic reviews are still conducted without rigor and report methodology in obscure language. Meta-analyses are also sometimes conducted inappropriately or use incorrect or questionable statistical methods.
• Systematic reviews may ask questions that hold little to no value for the practice of other researchers, clinicians, or practitioners. It has even been suggested that some reviews are conducted simply to boost an author’s publication numbers .

Current Climate

Taken together, the strengths and weaknesses of systematic reviews reveal an interesting tension at the root of evidence synthesis:

When conducted using appropriate methods, systematic reviews are a high-quality source of evidence-based answers. However, reviews take many months to complete and often struggle to keep pace with the rate at which new data is published.

We add to this the fact that dozens of reviews of varying methodological quality get through peer-review and get published every day. This leaves it up to the consumers (i.e., those busy academics, clinicians, and practitioners) to read each review’s methods carefully and decide if the data are trustworthy and the conclusions are sound.

The result is a perfect storm of wasted time, misinterpretation of findings, and missed opportunities for implementation of evidence-based practice. Despite all the great things that systematic reviews can do, there is simply too much out there (of variable quality) for the experts to properly analyze, digest, and reliably fold into their practice.

That’s All for Now…

This is a dower to end on, perhaps - but also represents a great opportunity. I know I am not alone in the belief that we can do better. Many notable efforts are underway to ensure quality evidence-based decisions are at the heart of practice. In our next post, we will explore some of these efforts and how they each aim to amplify the strengths of systematic reviews while addressing some weaknesses. In the end, I hope you will find a way in which you can help systematic reviews meet the needs, capacities, and realities of the modern biomedical research ecosystem.

Additional Resources and References

• Hoffmann F, Allers K, Rombey T, et al. Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000-2019. J Clin Epidemiol . 2021;138:1-11. doi:10.1016/j.jclinepi.2021.05.022
• Naudet F, Schuit E, Ioannidis JPA. Overlapping network meta-analyses on the same topic: survey of published studies. Int J Epidemiol . 2017;46(6):1999-2008. doi:10.1093/ije/dyx138
• Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q . 2016;94(3):485-514. doi:10.1111/1468-0009.12210

Latest Articles

Prism's essay title maker: the ultimate tool for crafting winning titles.

Prism's Book Name Generator: The Ultimate Tool for Authors

AI Research Tools: The Benefits of Using Prism's Cutting-Edge Technology

Understanding Large Perturbation Models

A brief, layperson's introduction to Large Perturbation Models (LPMs), a new tool in the drug development toolkit to simulate vast numbers of experiments digitally

Schedule a demo

Literature Review vs Systematic Review

• Literature Review vs. Systematic Review
• Primary vs. Secondary Sources
• Databases and Articles
• Specific Journal or Article

Subject Guide

Definitions

It’s common to confuse systematic and literature reviews because both are used to provide a summary of the existent literature or research on a specific topic. Regardless of this commonality, both types of review vary significantly. The following table provides a detailed explanation as well as the differences between systematic and literature reviews. 

Kysh, Lynn (2013): Difference between a systematic review and a literature review. [figshare]. Available at:  http://dx.doi.org/10.6084/m9.figshare.766364

• << Previous: Home
• Next: Primary vs. Secondary Sources >>
• Last Updated: Dec 15, 2023 10:19 AM
• URL: https://libguides.sjsu.edu/LitRevVSSysRev

• In the News
• Publications

• Research Methodology

Related Posts

• Ketika Cinta Ini Membunuhmu …
• Mind Map untuk Memahami Topik Penelitian
• Konsentrasi Game Technology
• Teknik Pengukuran Kualitas Perangkat Lunak

Systematic Literature Review: Pengantar, Tahapan dan Studi Kasus

Posted by Romi Satria Wahono on 15 May, 2016 in Research Methodology | 37 comments

Systematic literature review atau sering disingkat SLR atau dalam bahasa indonesia disebut tinjauan pustaka sistematis adalah metode literature review yang mengidentifikasi, menilai, dan menginterpretasi seluruh temuan-temuan pada suatu topik penelitian, untuk menjawab pertanyaan penelitian ( research question ) yang telah ditetapkan sebelumnya (Kitchenham & Charters, 2007). Metode SLR dilakukan secara sistematis dengan mengikuti tahapan dan protokol yang memungkinkan proses literature review terhindar dari bias dan pemahaman yang bersifat subyektif dari penelitinya. SLR adalah metode literature review yang biasa dilakukan peneliti di bidang farmasi dan kedokteran, meskipun boleh dikatakan baru mulai dibawa ke dunia computing  wa bil khusus software engineering pada tahun 2007 oleh Barbara Kitchenham lewat papernya berjudul Guidelines in performing Systematic Literature Reviews in Software Engineering .

Pengantar dan metode-metode yang ada untuk melakukan literature review sudah dibahas pada artikel sebelumnya tentang Literature Review: Pengantar dan Metode . Pada artikel ini, akan dibahas secara khusus dan mendetail tentang systematic literature review (SLR), beserta tahapan dan studi kasusnya. Untuk bisa memahami artikel ini dengan baik, direkomendasikan untuk mendownload dua dokumen di bawah:

• Systematic Literature Review (SLR) . Penjelasan komprehensif tentang metodologi penelitian dan systematic literature review. Contoh kasus utama mengikuti artikel ini, tapi ada contoh-contoh kasus bidang non computing.
• Systematic Literature Review: Pengantar, Tahapan dan Studi Kasus . Penjelasan komprehensif dari artikel ini, termasuk studi kasus tentang SLR mengikuti paper di bawah
• Romi Satria Wahono, A Systematic Literature Review of Software Defect Prediction: Research Trends, Datasets, Methods and Frameworks , Journal of Software Engineering, Vol. 1, No. 1, April 2015

Secara umum tahapan melakukan SLR terdiri dari 3 bagian besar: Planning , Conducting dan Reporting . Detail tiap tahapan seperti pada gambar di bawah.

Research Question (RQ) adalah bagian awal dan dasar berjalannya SLR. RQ digunakan untuk menuntun proses pencarian dan ekstraksi literatur. Analisis dan sintesis data, sebagai hasil dari SLR, adalah jawaban dari RQ yang kita tentukan di depan. RQ yang baik adalah yang bermanfaat, terukur, arahnya ke pemahaman terhadap state-of-the-art research dari suatu topik penelitian.

Formulasi RQ harus didasarkan pada lima elemen yang terkenal dengan sebutan PICOC:

• Population (P) : Target group dari investigasi
• Intervention (I) : Aspek detail dari investigasi, atau isu yang menarik bagi peneliti
• Comparison (C) : Aspek dari investigasi dimana Intervention (I) akan dibandingkan
• Outcomes (O) : Efek dan hasil dari Intervention (I)
• Context (C) : Setting dan lingkungan dari investigasi

Contoh PICOC dari paper SLR saya (Wahono, 2015) adalah seperti gambar di bawah.

Langkah berikutnya yang perlu kita lakukan adalah menyusun protokol SLR ( SLR Protocol ). Protokol SLR adalah rencana yang berisi prosedur dan metode yang kita pilih dalam melakukan SLR. Secara umum Protokol SLR biasanya memuat 7 elemen di bawah:

• Research Questions
• Search terms
• Selection criteria
• Quality checklist and procedures
• Data extraction strategy
• Data synthesis strategy

2. CONDUCTING

Tahapan conduting adalah tahapan yang berisi pelaksanaan dari SLR, dimana seharusnya sesuai dengan Protokol SLR yang telah kita tentukan. Dimulai dari penentuan keyword  pencarian literatur ( search string ) yang basisnya adalah dari PICOC yang telah kita desain di depan. Pemahaman terhadap sinonim dan alternatif pengganti kata akan menentukan akurasi pencarian literatur kita. Kemudian langkah berikutnya adalah penentuan sumber ( digital library ) dari pencarian literatur.  Karena literatur yang kita kumpulkan akan sangat banyak, mungkin ratusan atau ribuan paper, maka disarankan untuk menggunakan tool software untuk mempermudah kita mengelola literatur seperti Mendeley, Zotero, EndNote, dsb. Contoh strategi pemilihan literatur adalah seperti gambar di bawah.

Setelah semua literatur didapatkan, langkah berikutnya adalah memilih literatur yang sesuai. Untuk mempermudah proses ini kita direkomendasikan membuat kriteria yang berfungsi sebagai filter dalam pemilihan dan penolakan suatu literatur ( inclusion and exclusion criteria ). Contoh inclusion and exclusion criteria adalah seperti pada gambar di bawah.

Masih melanjutkan proses filtering dari literatur, selain inclusion and exclusion criteria , kita juga harus melakukan penilaian kualitas  ( quality assesment ) dari ratusan literatur yang kita temukan. Kitchenham et al. (2007) memberi rekomendasi bahwa penilaian kualitas literatur sebaiknya berdasarkan lima parameter di bawah:

• Apakah proses analisis data sudah tepat dilakukan?
• Apakah juga dilakukan analisis residual dan sensitifitas?
• Apakah akurasi statistik diambil dari data mentah?
• Seberapa baik komparasi metode yang dilakukan?
• Seberapa besar ukuran dari dataset yang digunakan dalam penelitian

Selain usulan Kitchengam di atas, peneliti lain menambahkan parameter penilaian kualitas supaya lebih akurat memfilter literatur. Misalnya (Salleh et al., 2011) menyatakan bahwa parameter yang sebaiknya digunakan untuk penilaian kualitas dari literatur adalah seperti pada gambar di bawah.

Langkah terakhir setelah kita mendapatkan literatur yang kita inginkan, adalah ekstraksi data ( data extraction ), kemudian melakukan sintesis berbagai hal yang kita temukan dari literatur-literatur yang sudah kita pilih ( synthesis of evidence ). Tujuan utama dari sintesis data adalah untuk menganalisis dan mengevaluasi berbagai hasil penelitian dari berbagai literatur, dan untuk memilih metode yang paling tepat untuk mengintegrasikan penjelasan dan interpretasi dari berbagai temuan tersebut (Cruzes & Dyba, 2011). Sintesis yang kita lakukan bisa berbentuk naratif atau kuantitatif ( meta analysis ). Langkah terakhir ini adalah langkah penting yang harus kita lakukan dengan detail dan hati-hati, karena kualitas SLR kita akan ditentukan dari hasil sintesis dan analisis yang kita lakukan.

3. REPORTING

Reporting  adalah tahapan penulisan hasil SLR dalam bentuk tulisan, baik untuk dipublikasikan dalam bentuk paper ke jurnal ilmiah atau untuk menyusun Bab 2 tentang Literature Review dari skripsi/tesis/disertasi kita. Struktur penulisan dari SLR biasanya terdiri dari 3 bagian besar, yaitu: Pendahuluan ( Introduction ), Utama ( Main Body ) dan Kesimpulan ( Conclusion ).  Bagian Pendahuluan akan berisi latar belakang dan landasan mengapa SLR pada suatu topik itu penting dan harus dilakukan. Sedangkan Bagian Utama akan berisi protokol SLR, hasil analisis dan sintesis temuan, serta diakhiri dengan diskusi yang membahas implikasi dari hasil SLR. Bagian Kesimpulan akan berisi rangkuman dari temuan yang kita dapatkan, sesuai dengan RQ yang kita tetapkan di depan.

Apabila SLR yang kita tulis akan dikirimkan dalam bentuk paper untuk suatu jurnal ilmiah, kita harus benar-benar menganalisis jurnal apa saja yang tepat untuk paper SLR kita. Tepat disini bisa bermakna dua, tepat dalam arti topiknya sesuai, dan juga tepat dalam arti bahwa kualitas temuan yang kita hasilkan dari SLR kita memang sesuai dengan level dari jurnal ilmiah yang akan kita pilih. Saya biasanya membuat list dari jurnal ilmiah yang saya targetkan untuk tempat publikasi, dan saya urutkan berdasarkan nilai SJR atau JIF dari jurnal tersebut. Saya juga berusaha mempelajari beberapa SLR yang sebelumnya dimuat pada jurnal-jurnal tersebut. Kemudian saya melakukan self-assesment apakah kualitas dari paper SLR saya sepadan dengan yang selama ini muncul di jurnal-jurnal tersebut. Gambar di bawah adalah contoh list jurnal-jurnal di bidang software engineering yang banyak memuat topik tentang SLR yang saya tulis yaitu software defect prediction. Kita akan lebih mudah mendapatkan list jurnal ini, karena proses SLR akan membawa kita ke paper-paper terbaik yang ada dalam suatu topik. Tinggal dilist saja di jurnal apa paper-paper pilihan kita itu diterbitkan.

Jujur tidak mudah melakukan tinjauan pustaka dan menulis paper dengan metode SLR. Disamping butuh waktu lama, juga butuh ketelatenan supaya kita bisa melakukan sintesis terhadap berbagai temuan dengan baik. Tapi paling tidak artikel ini dapat digunakan sebagai panduan praktis bagaimana tahapan melakukan SLR.

Sekali lagi Untuk bisa memahami artikel ini dengan baik, direkomendasikan untuk mendownload dua dokumen di bawah:

Tetap dalam perdjoeangan!

37 Comments

Mantab, pa…

Mohon izin menerapkan ilmu ini di penelitian tesis sy.

Monggo mas zaenal

menarik pak Romi, saya sedang proses untuk membuat SLR ini, saya dari ilmu-ilmu sosial meskipun demikian proses yang bapak jelaskan sepertinya bersifat umum artinya bisa digunakan untuk ilmu2 sosial dan eksak. Pertanyaan saya untuk tahap conducting apakah ada program software yang harus diinstal untuk bisa memilah jurnal yang kita inginkan secara otomatis? klo ada mohon petunjuk untuk mendapatkannya seperti apa ya? terimakasih.

Kayaknya SLR lebih rumit ya Pak

Lengkap nih penjelasannya, makasih pak

Perdjoeangan yang sesungguhnya akan segera dimulai 😀

Perdjoeangan yang sesungguhnya akan segera dimulai.

Nunut sinau pak.

Terimakasih pak..sangat mencerahkan saya yg sedang galau ^_^

Berapa lama waktu yang dibutuhkan untuk membuat sebuah paper dengan metode SYSTEMATIC LITERATURE REVIEW, Pak?

terima kasih banyak semoga bermanfaat

Terima kasih banyak atas penjelasannya. Alhamdulillah sudah ada bayangan mengenai SLR ini.

Assalamualaikum pak, saya mau tanya apakah pembuatan RQ itu boleh sedikit atau harus banyak?

terimakasih pak atas ilmu yang di berikan bapak sangat bermanfaat

maksasih pak, sangat membantu sekali berhubung skripsi saya harus ganti bikin slr karena sedang ada corona

terima kasih pak romi, izin copas yaaa 🙂

Terima kasih Pak… bermanfaat banget buat tugas review jurnal…

Semoga berkah ilmunya pak Romi, sangat membantu saya mengobati galau karena skripsi

Terima Kasih Banyak sharingnya, sangat bermanfaat dan saya ijin untuk mendownload. Tabarokallahu.

Terimakasih Ilmunya sangat bermanfaat di era Pandemi. Di tempat Saya Poltekkes Kemenkes menggunakan Meta analisis. Hampi mirip pak Ya. Mohon komentarnya…

Izin save pak. Thank you.

wah,, terimakasih pak artikelnya. bagi yang berminat membuat SLR Bersama sama, dibidang kesehatan, silahkan hubungi saya di email [email protected] ya,, terimakasih

Salam, selamat pagi Bapak. Terima kasih atas sharing ilmunya yang sangat mencerahkan. Semoga selalu diberkati dan bermanfaat. Aamiin.

Terima kasih ilmunya Pak, sangat membantu. Pengalaman pertama saya menyusun tugas akhir sebagai mahasiswa farmasi tanpa eksperimen lab, ini cukup menantang dan memberikan hikmah ilmu tersendiri buat saya.

selamat pagi, ijin bertanya…apakah sy bisa tahu perbedaan mendasar secara singkat antara scoping jurnal, sistematik review dan sistematika literatur review

kirim ke [email protected] terimakasih

sudah ada di materi mas .. monggo disimak dulu ya .. youtubenya juga ada

Good Articel, izin save ya pak

Mohon ijin save pak.Terima kash sangat membantu

Terima kasih penjelasannya pak. Sangat membantu dalam penulisan ilmiah, saya sedang mencoba menulis topik SLR menggunakan metode Prisma.

izin bertanya pak, untuk melakukan quality assessment terhadap jurnal yang kita teliti dengan menggunakan kriteria new castle ottawa scale, apakah hasil akhirnya dapat digabung dan ditambahkan kemudian dibagi dua? seperti misalnya jurnal A mendapatkan nilai 7 dari reviewer 1 dan mendapatkan nilai 5 dari reviewer 2. apakah hasil tersebut dapat digabung dan dibagi 2 pak? terima kasih banyak pak mohon bantuannya…

Apakah kantor brainmatics masih di menara Bidakara?

terimakasih pak

saya pernah membaca dalam proses SLR harus melibatkan reviewer lain untuk menyortir paper, apakah itu harus dilakukan atau boleh untuk tidak dilakukan? misalnya dalam rangka menulis skripsi atau tesis begitu

Terima kasih banyak Pak Romi ilmunya. Saya dapat banyak ilmu dari channel Pak Romi termasuk tentang SLR ini

Terima Kasih pak, saya sedang melakukan penelitian skripsi dengan menggunakan metode literature review/SLR dan diagram PRISMA, Atas informasi dan ilmu yang diberikan, saya menjadi tercerahkan.

Apakah benar bahwa dengan SLR pada akhirnya kita akan mendapatkan research gap, yang nantinya pada saat penulisan thesis/desertasi akan ditindak-lanjuti dengan mem-propose suatu metoda tertentu untuk mengatasinya ? Jika iya, apakah untuk mempertahankan research gap diharuskan menggunakan data empiris ? Demikian juga dengan hasil dari solusi yang di-propose ?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed .

My Categories

My schedule.

Copyright © 2004-2024 RomiSatriaWahono.Net. All rights reserved.

• Search Menu
• Sign in through your institution
• Advance articles
• Themed Collections
• Editor's Choice
• Ilona Kickbusch Award
• Supplements
• Author Guidelines
• Submission Online
• Open Access Option
• Self-Archiving Policy
• About Health Promotion International
• Editorial Board
• Advertising and Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, supplementary material.

• < Previous

Advertising appeals effectiveness: a systematic literature review

• Article contents
• Figures & tables
• Supplementary Data

Murooj Yousef, Sharyn Rundle-Thiele, Timo Dietrich, Advertising appeals effectiveness: a systematic literature review, Health Promotion International , Volume 38, Issue 4, August 2023, daab204, https://doi.org/10.1093/heapro/daab204

• Permissions Icon Permissions

Positive, negative and coactive appeals are used in advertising. The evidence base indicates mixed results making practitioner guidance on optimal advertising appeals difficult. This study aims to identify the most effective advertising appeals and it seeks to synthesize relevant literature up to August 2019. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework a total of 31 studies were identified and analyzed. Emotional appeals, theory utilization, materials, results and quality were examined. Across multiple contexts, results from this review found that positive appeals were more often effective than coactive and negative appeals. Most studies examined fear and humour appeals, reflecting a literature skew towards the two emotional appeals. The Effective Public Health Practice Project framework was applied to assess the quality of the studies and identified that there remains opportunity for improvement in research design of advertising studies. Only one-third of studies utilized theory, signalling the need for more theory testing and application in future research. Scholars should look at increasing methodological strength by drawing more representative samples, establishing strong study designs and valid data collection methods. In the meantime, advertisers are encouraged to employ and test more positive and coactive advertising appeals.

Advertising appeals have witnessed an increase in research interest and scholarly attention in recent years. Studies investigate appeal effectiveness [e.g. ( Jordan et al. , 2015 ; Lee, 2018 )] and to a lesser extent systematic and meta-analytic studies attempting to synthesize results are evident ( O’Keefe and Jensen, 2009 ; Jenkin et al. , 2014 ; Hornik et al. , 2016 ). These studies however are limited in their focus [e.g. fear appeals ( Tannenbaum et al. , 2015 ; Esrick et al. , 2019 )], context [e.g. disease detection behaviours ( O’Keefe and Jensen, 2009 )], media type, [e.g. mass media ( Elder et al. , 2004 )] and comparison of general advertising appeal types [e.g. rational vs. emotional (fear and humour) vs. metaphor appeals ( Hornik et al. , 2017 )]. Taken together, a review of the literature indicates clear gaps requiring an evidence review focussed on synthesizing studies seeking to examine positive versus negatively framed advertising appeal effectiveness that are context free, not media specific, includes rational as well as emotional studies of different emotional valances (positive, negative and coactive), and extends the range of emotions examined beyond fear and humour which is heavily investigated in the literature. Given that negatively framed appeals dominate behaviour change and prevention studies, a systematic literature review that explores the effectiveness of different advertising approaches is important, timely and called for [e.g. ( Williams et al. , 2004 ; Armstrong, 2010 ; Hornik et al. , 2016 )].

Hornik et al. (Hornik et al. , 2016 ) based their meta-analytic review on rational, emotional (i.e. fear, humour and sex) and metaphor advertising appeals, limiting their results to specific appeal types. The current study seeks to build on their study, extending investigation to other appeals (e.g. coactive) to ascertain the extent these have been used effectively to deliver behaviour change. Following Hornik et al. (Hornik et al. , 2016 ), we argue that positive emotional advertising appeals are more effective in changing behaviour than negative and rational advertising appeals. However, in contrast to their study, we do not follow their general classification of appeals (i.e. rational, emotional and metaphor), but rather we include a wider set of studies that look at rational, emotional, positive, negative and coactive advertising appeals in different campaign contexts (e.g. social and commercial).

Advertising appeals

An advertising appeal refers to the use of persuasion strategies to attract attention, create relevance and memorability, raise awareness and induce action ( Armstrong, 2010 ). An advertising message can appeal to one’s cognition (i.e. rational appeals), emotions (i.e. emotional appeals) or both. Rational appeals rely on arguments, reason and facts to create persuasion ( Dahlen et al. , 2010 ). In contrast, emotional appeals seek to induce certain emotions in the audience to make the message memorable and more persuasive to take action ( Dahlen et al. , 2010 ). The emotional versus rational debate has been widely discussed with scholars exploring effectiveness in different advertising aims, contexts, business types and target audiences [see, e.g. ( Mattila, 1999 ; Matthes and Wonneberger, 2014 ; Akpinar and Berger, 2017 ; Moran and Bagchi, 2019 )]. Two recent meta-analytic studies identified that consumers respond more favourably to emotional appeals than they do to rational appeals ( Hornik et al. , 2016 , 2017 ).

Effectiveness of different emotional appeals utilized in advertising messages has also received attention. Emotional appeals can be classified as positive, negative or coactive based on the valance of emotion employed. Each emotional valence exerts different effects on judgement and therefore affects perceptions and behaviours differently ( Lerner and Keltner, 2000 ). The literature reports mixed results for advertising effectiveness when it comes to positive versus negative emotional appeals. For example, while fear appeals were found to generate defensive reactions ( Witte and Allen, 2000 ) and result in a boomerang effect for young adults ( Lennon et al. , 2010 ), other studies found negative appeals to be effective in creating behaviour change when compared to positive and neutral appeals ( Struckman‐Johnson et al. , 1994 ; Small and Verrochi, 2009 ; Tay, 2011 ; Sun, 2015 ). Neutral appeals are discussed mainly in charity advertising [see, e.g. ( Small and Verrochi, 2009 )], where positive and negative appeals are compared to neutral (no emotion) ads.

Positive emotional appeals are explored in the literature to a lesser extent reflecting their limited use in advertising campaigns focussed on health prevention and related contexts ( Tay, 2005 ; Dunstone et al. , 2017 ). Inducing positive emotions through advertising messages was found to yield more positive attitudes to the advertisement ( Lau-Gesk and Meyers-Levy, 2009 ), higher liking of the message ( Hornik et al. , 2017 ) and a stronger impact on behaviour than negative emotional appeals in multiple contexts such as safe driving ( Plant et al. , 2017 ), reducing binge drinking among college students ( Lee, 2018 ), encouraging environmental friendly behaviour ( Wang et al. , 2017 ; Skurka et al. , 2018 ), health behaviour ( Jordan et al. , 2015 ; Vaala et al. , 2016 ) and anti-cyber bullying ( Alhabash et al. , 2013 ). However, positive emotional appeals were found to be less effective for highly involved consumers ( Yoon and Tinkham, 2013 ) and for female audiences ( Noble et al. , 2014 ) when compared to low involved and male audiences respectively.

Recently there has been an interest in the literature in the use of coactive emotional appeals that seek to induce both positive and negative emotions simultaneously ( Nabi, 2015 ; Yoon, 2018 ). It is hypothesized that the use of a threat-relief emotional message by combining emotions like fear and humour will result in a stronger persuasion outcome ( Nabi, 2015 ). Positive emotions have the ability to reduce the defensive reactions that negative appeals generate, making them more effective in changing behaviour ( Mukherjee and Dubé, 2012 ; Bennett, 2015 ). Eckler and Bolls and Alhabash et al. found coactive appeals ( Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ) to have a stronger impact than negative appeals but their work also indicated that coactive appeals are weaker than positive appeals. No known systematic or meta-analytic review has synthesized the effectiveness of coactive advertising appeals, signalling the need for a review study.

Emotion can be defined as the psychological reaction to an event, a memory and specific types of media ( Allen et al. , 2005 ). Emotions are usually provoked by an internal stimulus that generates a strong short-term reaction influencing one’s attitudes towards something ( Scherer, 2005 ). Wu et al. report that being exposed to an advertisement ( Wu et al. , 2018 ), even a very short exposure, will induce both strong and weak emotions. The type of emotions used in an advertisement will have different results for the audience. Using neural signal tools like heartrate monitors, Kaye et al. (Kaye et al. , 2016 ) found that negative advertisements stimulate respondents while positive advertisements result in a more relaxed feeling.

RQ1. Which emotional advertising appeal is more effective in creating behaviour change across different contexts?

There is no recent systematic review that looks beyond the context of advertising (e.g. health) and valance of emotions (e.g. fear appeals) to understand the effectiveness of positive versus negative and coactive advertising appeals. The aims of this systematic review study are two-fold. First, to highlight the most effective advertising appeal based on empirical research findings utilizing behavioural (e.g. driving speed) or behavioural proxy (e.g. intentions) measures up to August 2019. Second, this review analyses the quality of published studies in the field based on the Effective Public Health Practice Project (EPHPP) protocol to guide future research.

emotional appeals or emotion* based advertis* AND appeal* AND advertis* or public service announcement or psa or message or communication or strategy or promot* or campaign or experiment

In total, 2384 records were initially identified (see Figure 1 for a flowchart of the search process adopted). Due to the magnitude and focus of each database and its alignment with the search terms, there was variance in the number of records produced from each database. The downloaded records were collated using Endnote. First, all duplicate records were removed leaving a total of 1507 unique records. Second, unqualified records including conference and government reports, unidentifiable full text, as well as records not in English were removed. Finally, titles and abstracts of remaining records were assessed and classified into the exclusion criteria categories: studies using non advertising materials (e.g. news articles), non-emotional based advertising, non-experimental studies (e.g. content analysis and literature reviews), studies exploring only one type of appeal (i.e. negative, positive, mixed or rational), rational versus emotional appeal studies, message framing studies (e.g. gain vs. loss frame), studies lacking behaviour or intention measures of effectiveness.

Systematic search diagram using PRISMA process.

After application of the exclusion criteria, a total of 25 articles undertaking a direct comparative evaluation of the effectiveness of positive and negative appeals were identified. Next, backward and forward searching using authors’ names, Google Scholar and reference lists were completed. A further six articles were identified. In total, 31 articles were analysed. The full list of papers can be found in Supplementary Appendix A .

Data extraction and analysis

The included studies were analysed in terms of (i) the employed materials, (ii) study characteristics and results and, (iii) study quality.

Employed materials and media

Each study’s stimulus was screened to determine the type of media (e.g. video, print, audio), the type of emotion (e.g. fear, guilt, happiness), the target issue (e.g. health behaviour, safe driving, environmental behaviour) and the type of appeals tested (e.g. positive, negative, coactive, rational appeals). This categorized studies based on the type of stimulus used to identify patterns and examine appeal effectiveness.

Study characteristics and results

The 31 identified studies were analysed based on their sample size, sample characteristics (e.g. age and gender), data collection methods (e.g. self-report or objective measures), data collection time points (e.g. post exposure only, pre and post exposure or after a delayed period of time), the employed theory (if any) and mediators and moderator measures of effectiveness. Study outcome measures that were set to warrant inclusion in the review were restricted to behaviour or behavioural intention measures. Studies were excluded if an outcome evaluation was not undertaken to examine advertising effectiveness. For included studies, results were categorized based on the most effective appeal, namely (i) positive, (ii) negative, (iii) no difference/inconclusive or (iv) mixed if positive and negative appeals were found to be effective for different cohorts.

Quality assessment

The quality of the included studies was assessed using the EPHPP quality assessment tool for quantitative studies ( Effective Public Health Practice Project, 2019 ). The EPHPP tool is suitable for evaluating multiple study designs ( Deeks et al. , 2003 ) and has been used to assess the quality of advertising studies in previous reviews ( Becker and Midoun, 2016 ). The assessment tool is valid ( Thomas et al. , 2004 ; Jackson and Waters, 2005 ) and suitable for use in systematic reviews examining effectiveness ( Deeks et al. , 2003 ). Each study was rated using six EPHPP criteria: (i) selection bias—how representative the sample is of the target population; (ii) study design—the likelihood of bias due to the allocation process in the study; (iii) confounders—the extent to which groups were balanced at baseline with respect to confounding variables; (iv) blinding—whether participants were aware of the study objectives and researchers participating in the study were aware of each group participation status; (v) data collection—whether study measures were valid and reliable and (vi) withdrawals and drop outs—the percentage of participants remaining in the study at the final data collection period in all groups ( Thomas et al. , 2004 ). Each individual aspect is rated as weak, moderate or strong and an overall rating is applied to each study ( Thomas et al. , 2004 ). All studies assessed through the EHPHH tool were rated by two researchers and inter-reliability scores exceeded the 80% threshold. Discrepancies were discussed and resolved with all three authors.

Description of included studies

In total, 31 studies qualified for inclusion. More than half of studies were from the United States [ n = 18; e.g. ( Alhabash et al. , 2013 ; Bleakley et al. , 2015 )], followed by Australia [ n = 5; e.g. ( Noble et al. , 2014 ; Kaye et al. , 2016 )], the rest ( n = 8) were from Canada ( Tay, 2011 ), United Kingdom ( Eckler and Bolls, 2011 ), Germany ( Jäger and Eisend, 2013 ), Belgium ( Faseur and Geuens, 2010 ), Netherlands ( Hendriks et al. , 2014 ), China ( Wang et al. , 2017 ), Taiwan ( Wu et al. , 2018 ) and South Korea ( Sun, 2015 ) (see Figure 2 for study locations). Most studies addressed social issues ( n = 28) such as safe driving ( Lewis et al. , 2008 ; Taute et al. , 2011 ; Tay, 2011 ; Jäger and Eisend, 2013 ; Previte et al. , 2015 ; Kaye et al. , 2016 ; Plant et al. , 2017 ), charity donations ( Small and Verrochi, 2009 ; Faseur and Geuens, 2010 ; Kemp et al. , 2013 ; Cao and Jia, 2017 ; Zemack-Rugar and Klucarova-Travani, 2018 ), health ( Struckman‐Johnson et al. , 1994 ; Lee and Ferguson, 2002 ; Passyn and Sujan, 2006 ; Hendriks et al. , 2014 ; Bleakley et al. , 2015 ; Jordan et al. , 2015 ; Vaala et al. , 2016 ; Thainiyom and Elder, 2017 ; Lee, 2018 ), the environment ( Yoon and Tinkham, 2013 ; Noble et al. , 2014 ; Wang et al. , 2017 ; Skurka et al. , 2018 ), organ donation ( Rodrigue et al. , 2014 ; Sun, 2015 ) and cyberbullying ( Alhabash et al. , 2013 ). Three studies were undertaken in commercial settings with authors examining toothbrush, influenza vaccine, alcohol, cars and insurance advertisements ( Brooker, 1981 ; Eckler and Bolls, 2011 ; Wu et al. , 2018 ) (see Figure 3 ).

Location of included studies.

Studies by targeted issue.

Most studies looked at positive versus negative advertising appeals [ n = 19; e.g. ( Kemp et al. , 2013 ; Kaye et al. , 2016 )], only two studies included positive, negative and coactive appeals ( Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ), while the rest incorporated a rational [ n = 8; e.g. ( Sun, 2015 ; Skurka et al. , 2018 )] or neutral appeal [ n = 2 ( Small and Verrochi, 2009 ; Zemack-Rugar and Klucarova-Travani, 2018 ); see Figure 4 ] in their tests. In terms of emotions, fear versus humour was most frequently examined with 12 (38%) studies comparing the two emotions [e.g. ( Tay, 2011 ; Vaala et al. , 2016 )]. Of all tested emotional appeals, fear was the most studied appeal (48%) followed by humour (45%). Positive emotions such as pride ( Kemp et al., 2013 ; Noble et al. , 2014 ; Wang et al. , 2017 ), hope ( Rodrigue et al. , 2014 ; Thainiyom and Elder, 2017 ), love ( Previte et al. , 2015 ) and a range of negative emotions such as disgust ( Hendriks et al. , 2014 ), anger ( Rodrigue et al. , 2014 ), shame ( Previte et al. , 2015 ), regret ( Taute et al. , 2011 ) and guilt ( Noble et al. , 2014 ) were also considered. Seven studies did not specify which positive and negative emotions were tested ( Faseur and Geuens, 2010 ; Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ; Sun, 2015 ; Kaye et al. , 2016 ; Plant et al. , 2017 ; Wu et al. , 2018 ).

Studies by the tested appeals.

Only three studies utilized objective data collection tools. Objective outcome data included GPS speed trackers ( Kaye et al. , 2016 ), driving stimulators ( Plant et al. , 2017 ) and donation amounts ( Small and Verrochi, 2009 ). The rest of the studies relied on self-reported measures [ n = 28; e.g. ( Jäger and Eisend, 2013 ; Skurka et al. , 2018 ; Wu et al. , 2018 )]. The majority of studies ( n = 24) collected data post exposure only [e.g. ( Taute et al. , 2011 ; Sun, 2015 )]. Four studies included a post exposure and a follow-up data collection time point after a delayed period of time ( Passyn and Sujan, 2006 ; Lewis et al. , 2008 ; Hendriks et al. , 2014 ; Plant et al. , 2017 ). Two studies collected data pre and post exposure ( Rodrigue et al. , 2014 ; Previte et al. , 2015 ) and only one study collected data at pre, post and follow-up time points ( Kaye et al. , 2016 ).

Only 35% of studies were guided by theories. Theories that were reported included the Elaboration Likelihood Model ( Lewis et al. , 2008 ), Extended Parallel Process Model ( Tay, 2011 ), Theory of Planned Behaviour ( Hendriks et al. , 2014 ), Affect as Information Theory ( Taute et al. , 2011 ) and other theories (see Supplementary Appendix A ).

Study outcomes

The aim of this systematic review was to highlight effective advertising appeals. This is based on the ability of the appeal to influence behaviour or behavioural intentions significantly ( P < 0.05) in the desired direction (e.g. reduce drink driving). The results of the 31 included studies indicate that positive advertising appeals are slightly more effective than negative and coactive advertising appeals. It is important to note there is evidence of effectiveness for all appeal types and each context and target audience differ in appeal effectiveness requiring pre-testing and examination prior appeal consideration. Thirty-five per cent ( n = 11) of studies reported positive appeals to be more effective ( Brooker, 1981 ; Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ; Rodrigue et al. , 2014 ; Previte et al. , 2015 ; Sun, 2015 ; Plant et al. , 2017 ; Wang et al. , 2017 ; Lee, 2018 ; Wu et al. , 2018 ; Zemack-Rugar and Klucarova-Travani, 2018 ), while 26% ( n = 8) reported negative appeals to have a stronger persuasion effect than positive appeals ( Struckman‐Johnson et al. , 1994 ; Small and Verrochi, 2009 ; Tay, 2011 ; Hendriks et al. , 2014 ; Noble et al. , 2014 ; Bleakley et al. , 2015 ; Kaye et al. , 2016 ). Nineteen per cent of studies ( n = 6) showed mixed results. Where mixed results were reported the mixed outcomes occurred as a result of range of factors including gender ( Kemp et al. , 2013 ; Thainiyom and Elder, 2017 ), connection to others ( Faseur and Geuens, 2010 ), prior attitudes ( Jäger and Eisend, 2013 ), time of assessment after exposure ( Lewis et al. , 2008 ), issue involvement ( Yoon and Tinkham, 2013 ) and psychological involvement ( Cao and Jia, 2017 ). Five studies (16%) did not find any significant differences in effectiveness between positive and negative appeals ( Passyn and Sujan, 2006 ; Thainiyom and Elder, 2017 ; Skurka et al. , 2018 ). Finally, only one study reported inconclusive results due to unrepresentative sample ( Lee and Ferguson, 2002 ). Figure 5 showcase results of the included studies.

Results supporting different appeals effectiveness or reporting mixed, indifferent or inconclusive results.

A quality assessment of the identified papers was conducted using the EPHPP tool (see Supplementary Appendix B ). Of the 31 included studies, 26 were assessed as weak in the global rating, five were assessed as moderate and none were assessed as strong. Selection bias was likely in many studies due to the use of student samples or bias to a geographical area. Only one study was somewhat likely to have a representative sample ( Skurka et al. , 2018 ). Five studies included a control group and randomly allocated participants into experimental groups (e.g. positive and negative stimuli) therefore these were assessed as strong in terms of study design ( Struckman‐Johnson et al. , 1994 ; Bleakley et al. , 2015 ; Jordan et al. , 2015 ; Vaala et al., 2016 ; Skurka et al. , 2018 ). Six were assessed as moderate ( Lewis et al. , 2008 ; Hendriks et al. , 2014 ; Previte et al. , 2015 ; Sun, 2015 ; Kaye et al. , 2016 ; Plant et al. , 2017 ), while the rest ( n = 20) were weak due to their cross sectional nature [e.g. ( Alhabash et al. , 2013 ; Jäger and Eisend, 2013 )].

In terms of confounders, about one-third of studies ( n = 10, 32%) reported either no baseline differences between groups or controlled for at least 80% of relevant confounders resulting in a strong rating. The rest of the studies ( n = 21) did not report potential confounders or account for confounds during analysis and were therefore assessed as weak [e.g. ( Alhabash et al. , 2013 ; Bleakley et al. , 2015 ; Lee, 2018 )]. Only two studies (10%) clearly reported that both the assessors and participants were not blinded in the experiment resulting in a weak rating ( Rodrigue et al. , 2014 ; Plant et al. , 2017 ). The rest of the studies ( n = 29, 87%) were rated as moderate as it was not clear if the participants and assessors were blinded or not. In terms of data collection methods, over half of the included studies ( n = 19, 61%) did not provide evidence of the validity of the reported measures and were therefore assessed as weak.

Two studies were assessed as moderate in their data collection method as they reported on validity but not reliability of the measures ( Jordan et al. , 2015 ; Plant et al. , 2017 ), while the rest ( n = 10, 32%) were rated strong for providing evidence of the validity and reliability of the reported outcomes measures [e.g. ( Kemp et al. , 2013 ; Noble et al. , 2014 ; Sun, 2015 ; Kaye et al. , 2016 )]. For the retention rate of participants, only two programs were assessed as strong with more than 80% completing the experiment ( Kaye et al. , 2016 ; Plant et al. , 2017 ). The rest were rated as moderate due to the lack of retention rate reporting [e.g. ( Jäger and Eisend, 2013 )], low completion rate [e.g. ( Rodrigue et al. , 2014 ; Jordan et al. , 2015 )] or due to the post exposure nature of studies where retention rate is not applicable [e.g. ( Faseur and Geuens, 2010 )].

The aims of this study were two-fold. This study aimed to identify which appeal type (positive, negative and/or coactive) was most likely to change social and commercial behaviour and to assess the quality of studies reported in peer review literature. This is the first known systematic review that is not limited to an emotion, appeal type, context or media. Our findings extend understanding in three key ways. First, this article extends understanding of appeal effectiveness with consideration of the effectiveness of coactive appeals. Second, it examines the extent of theory and emotion use in the included studies. Third, it assesses study quality identifying how researchers can enhance the evidence base by improving study quality.

Positive, negative or coactive?

Consistent with the literature ( Jenkin et al. , 2014 ; Hornik et al. , 2016 ) our findings confirm a slight persuasive advantage of positive advertising appeals over negative appeals. Positive appeals are able to increase consumers’ perceived response efficacy more than negative appeals ( Zemack-Rugar and Klucarova-Travani, 2018 ); help consumers realize the rewards of the promoted behaviour [e.g. moderate alcohol consumption ( Previte et al. , 2015 )]; induce positive attitudes—more than negative and coactive appeals—and therefore affect behavioural intentions positively ( Eckler and Bolls, 2011 ; Wang et al. , 2017 ). According to studies synthesized in the present review, positive appeals yield higher acceptance of the advertising message ( Alhabash et al. , 2013 ) by creating a positive climate in which messages may be received ( Brooker, 1981 ), reducing reactance [e.g. skipping, ignoring, backlash or resisting ( Wu et al. , 2018 )] and increasing message liking ( Lee, 2018 ). Further outcomes accruing from positive appeals include illustration of positive benefits of the promoted behaviour by inducing empathy and reducing guilt ( Rodrigue et al. , 2014 ).

Negative appeals dominate social change practice and while evidence for effectiveness exists, there appears to be less support in comparison to positive and coactive appeals based on this study’s findings. Mixed results were also evident in other studies. For example, Kemp et al. argued that positive appeals are ( Kemp et al. , 2013 ) more persuasive with a male audience than a female audience, while Jäger and Eisend found participants with less ( Jäger and Eisend, 2013 ) favourable prior attitudes produce higher change in intentions to drink drive when exposed to positive emotional appeals.

The effectiveness of coactive appeals compared to single appeals was examined by 2 of the 31 included studies. Their findings suggest coactive appeals are less effective than positive appeals and more effective than negative appeals ( Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ). Positive appeals require less cognitive processing, generate a general sense of pleasantness, are more likable and facilitate positive attitudes towards the advertisement making the advertised behaviour more appealing and taking action more tempting. On the contrary, the more negative an ad is, the less likable it is and the less likely viewers are to take action (i.e. share on social media). Therefore, coactive emotional appeals come in the middle, they are more effective than negative appeals but less effective than positive appeals ( Alhabash et al. , 2013 ). Interestingly, the two studies including coactive appeals in their experiments focused on viral sharing behaviour. Taking the target behaviour in consideration, their results can be interpreted more specifically. Previous studies found both emotional valence and arousal to affect content sharing and virality of advertisements ( Berger, 2011 ; Berger and Milkman, 2012 ). More specifically, content that are emotionally arousing (either positive or negative) are more likely to be shared with others than those less arousing. Furthermore, ads that are more positive in nature are more likely to be shared than negative ads ( Berger and Milkman, 2012 ). Moreover, the use of positive emotions along with negative emotions helps reduce the defensive responses of the audience resulting in a higher persuasion effect ( Mukherjee and Dubé, 2012 ). Hence, the studies included in this systematic review found coactive appeals to be more effective than negative appeals. When testing behaviour beyond sharing and virality, Yousef et al (2021) found positive appeals and coactive appeals to have similar effect on behaviour. Target audience plays a role in different appeals effectiveness, including coactive appeals. Studying advertising effect on young adults road safety perceptions and behaviour intentions, Yousef et al (2021) found coactive appeals to be more effective than single emotional appeals. The limited and mixed evidence for coactive appeals effectiveness is mainly due to the limited studies including such appeals in their experiments. More evidence is needed to determine coactive appeals effectiveness in other contexts and behaviours.

Applying theories and moving beyond fear and humour appeals

Over the years, advertising researchers have been under pressure to deliver relevant and practical findings that practitioners can follow and utilize ( Pitt et al. , 2005 ). It is argued that advertising research has formulated theories with ‘a high level of generality’ which makes them difficult to apply in practice ( Cornelissen and Lock, 2002 ). As a corollary, and due to the empirical nature of the included studies, these issues may have led to the limited application of theoretical frameworks. Pitt et al. (Pitt et al. , 2005 ) found only a minority of papers published in an 11-year period made explicit use of theories. Our findings confirm their research with more than half of the included studies lacking a theoretical base. Examples exist indicating how and where theory has been applied by researchers in intervention design, recruitment, implementation and evaluation [see ( Willmott et al. , 2019 )]. For example, Wadsworth and Hallam (Wadsworth and Hallam, 2010 ) applied social cognitive theory to an e-communication intervention identifying which theoretical constructs led to a physical activity increase. Theory did not only inform their study but was tested, refined and built on by the authors. This type of theory application can enhance study outcomes, better inform future research and systematically identify which theories are effective and for which audiences ( Willmott et al. , 2019 ).

Similarly, limited studies explored emotions beyond the heavily investigated emotions of fear and humour. Little is known about how other emotions effectiveness such as anger, disgust, guilt, love, joy and pride appeals deliver (or not) behavioural change. This reinforces past studies which have identified the limited use of emotions in advertising messages ( Tay, 2005 ; Dunstone et al. , 2017 ), not because other emotions are less effective but because there is limited evidence of effectiveness. When studies explore more emotions, new evidence emerges enabling practitioners to innovate and capture the attention of their audience. For example, Previte et al. ( Previte et al. , 2015 ) found a persuasive advantage for love and happiness (two emotions that are rarely examined in the advertising literature) over fear and shame appeals in moderate drinking advertising message, highlighting the potential of other emotions to yield desired results.

Enhancing study rigour to deliver a stronger evidence base for advertising effectiveness

Study quality assessment frameworks provide tools to assess the quality of research. The stronger the study, the more the policy, practitioner and research community can rely on the study findings. This study applied the EPHPP quality assessment tool ( Effective Public Health Practice Project, 2019 ) to assess study quality. Of particular concern is that no one study overall was rated as strong in the current review. In general, the methodological quality of the included studies was low. In the absence of strong evidence any conclusions drawn in the present evidence review and earlier meta-analytic and systematic literature reviews should be interpreted with caution until stronger study designs emerge. Within the present review notable, methodological problems included selection biases, weak study designs and invalid data collection methods.

A common issue with sampling is the use of student samples and samples from a specific region for convenience, resulting in selection bias. While calls for adoption of probability sampling procedures in the academic literature have been made ( Plant et al. , 2011 ; Sarstedt et al. , 2018 ), limited adoption of non-probability sampling is evident. In the absence of replication across samples or regions his reduces the generality of these studies making them bound to their sample and regional characteristics. Furthermore, the use of cross-sectional study designs contributed to the overall weak rating for most studies in this review. Including only a post-test immediately after exposure to the tested advertisements can lead to different result compared to testing over a delayed period of time ( Lewis et al. , 2008 ) making the results incomplete and the findings less comprehensive. Researchers are encouraged to include more than one time point for data collection to measure behaviour change over time. Finally, the validity and reliability of data collection methods used in the included studies are mostly weak. This is a reflection of the limited use of theories, with more studies bringing in their own measures without testing their validity or reliability before conducting their evaluations. Future research should focus on increasing the validity of their studies by utilizing previously validated measures from the literature ( David and Rundle-Thiele, 2018 ). This makes the study easier to replicate and its findings more reliable. Taken together, future research should aim to address these issues and improve the methodological quality of advertising evaluation studies to enhance empirical evidence.

Limitations

This study is restricted by several important limitations, which should be considered when interpreting the findings. First, the study is limited by the search parameters utilized and the study quality frameworks applied. For example, the review only includes studies that empirically test advertising appeal effectiveness (positive, negative, coactive), using behavioural measures (e.g. purchase intentions) that have been published in peer-reviewed English literature. Hence studies that rely on other measures (e.g. attitudes) or evaluate other message tactics (e.g. framing) and non-English and non-peer-reviewed studies, were excluded. Grey literature may contribute important information and future studies may benefit from examining these sources. The study focused mainly on emotional appeals, hence rational appeals were not included. Future reviews should compare rational and emotional appeals for more comprehensive findings. Second, due to the heterogeneity in the tested appeals, study populations and reporting of results, a meta-analysis was not possible, and a qualitative description of study outcomes was provided. Few studies included effect sizes and odds ratios, limiting our ability to compare effectiveness for the different advertising appeals. Third, results of the current review are collected from different contexts and behaviours and generalization of findings cannot be extended beyond this review. Moreover, pre-tests should be carried out before adopting any advertising appeal for any specific context, behaviour and target audience. Finally, based on the quality assessment of the included studies there is a clear absence of strong rigour experiments, hence any conclusions drawn in the present review should be interpreted with caution.

Future research

Future research should examine appeals effectiveness by utilising and applying advertising theories, investigating emotions beyond fear and humour in advertising appeals, increase the strength of their studies by following EPHPP guidelines, or other study quality frameworks, to design rigorous experiments and ensure that valid replicable analysis is reported. More effort should be made to draw representative samples, ensuring valid data collection methods and designing strong experiments that test effectiveness pre, post and after a delayed period of time following exposure. Furthermore, more studies should include coactive appeals in their evaluations to confirm their effectiveness compared to single appeal use as only a limited number of studies explored this type of appeal.

Future systematic literature reviews should build on this study by including other advertising tactics such as non-emotional appeals and gain and loss framing which can provide a wider picture of advertising effectiveness. Moving forward, consensus on advertising effectiveness outcome measures should be generated by the advertising research community. By agreeing on standard outcome measures, as occurs in tobacco control research, the research community could then advance understanding further via meta-analyses. Any effort that can reduce data transformation practices will serve to ensure synthesis studies can advance knowledge through delivery of the highest quality research that can inform policy and advertising practices.

This systematic review examined advertising appeals effectiveness based on the literature up to August 2019. Our findings support previous meta-analytic reviews in confirming positive appeals effectiveness over negative appeals. We extend on their findings however by including coactive advertising appeals. Across different contexts and behaviours, this review found positive appeals to be effective more often than negative appeals and coactive appeals. When all three appeals are studied, evidence suggest coactive appeals are more effective than negative appeals and less effective than positive appeals. Specifically, this review highlighted the scarce of theory use in advertising research signalling the need for more attention to embed theory into advertising design and evaluation. Moreover, a major concern raised by this review is the quality of the published papers. A greater focus should be made by authors to utilize valid data collection methods, representative samples and strong study designs. This research has contributed to a better understanding of advertising appeal effectiveness and may be of interest to policy makers, advertising professionals and designers and researchers who are interested in maximizing return on investment.

Supplementary material is available at Health Promotion International online.

Akpinar E. , Berger J. ( 2017 ) Viral marketing works best with emotional appeals . Journal of Marketing Research , 54 , 318 – 330 .

Google Scholar

Alhabash S. , McAlister A. R. , Hagerstrom A. , Quilliam E. T. , Rifon N. J. , Richards J. I. ( 2013 ) Between likes and shares: effects of emotional appeal and virality on the persuasiveness of anticyberbullying messages on Facebook . Cyberpsychology, Behavior and Social Networking , 16 , 175 – 182 .

Allen C. T. , Machleit K. A. , Schultz Kleine S. , Sahni Notani A. ( 2005 ) A place for emotion in attitude models . Journal of Business Research , 58 , 494 – 499 .

Armstrong J. S. ( 2010 ) Persuasive Advertising: Evidence-Based Principles . Palgrave Macmillan, London .

Google Preview

Becker S. J. , Midoun M. M. ( 2016 ) Effects of direct-to-consumer advertising on patient prescription requests and physician prescribing: a systematic review of psychiatry-relevant studies . The Journal of Clinical Psychiatry , 77 , e1293 – e1300 .

Bennett R. ( 2015 ) Individual characteristics and the arousal of mixed emotions: consequences for the effectiveness of charity fundraising advertisements . International Journal of Nonprofit & Voluntary Sector Marketing , 20 , 155 – 209 .

Berger J. ( 2011 ) Arousal increases social transmission of information . Psychological Science , 22 , 891 – 893 .

Berger J. , Milkman K. L. ( 2012 ) What makes online content viral? Journal of Marketing Research , 49 , 192 – 205 .

Bleakley A. , Jordan A. B. , Hennessy M. , Glanz K. , Strasser A. , Vaala S. ( 2015 ) Do emotional appeals in public service advertisements influence adolescents’ intention to reduce consumption of sugar-sweetened beverages? Journal of Health Communication , 20 , 938 – 948 .

Brooker G. ( 1981 ) A comparison of the persuasive effects of mild humor and mild fear appeals . Journal of Advertising , 10 , 29 – 40 .

Cao X. , Jia L. ( 2017 ) The effects of the facial expression of beneficiaries in charity appeals and psychological involvement on donation intentions . Nonprofit Management and Leadership , 27 , 457 – 473 .

Cornelissen J. P. , Lock A. R. ( 2002 ) Advertising research and its influence on managerial practice . Journal of Advertising Research , 42 , 50 – 55 .

Dahlen M. , Lange F. , Smith T. ( 2010 ) Marketing communications: a brand narrative approach. John Wiley & Sons, United States, New Jersey.

David P. , Rundle-Thiele S. ( 2018 ) Social marketing theory measurement precision: a theory of planned behaviour illustration . Journal of Social Marketing , 8 , 182 – 201 .

Deeks J. J. , Dinnes J. , D’Amico R. , Sowden A. J. , Sakarovitch C. , Song F. , et al. ; European Carotid Surgery Trial Collaborative Group . ( 2003 ) Evaluating non-randomised intervention studies . Health Technology Assessment (Winchester, England) , 7 , iii, 1 – 173 .

Dunstone K. , Brennan E. , Slater M. D. , Dixon H. G. , Durkin S. J. , Pettigrew S. et al.  ( 2017 ) Alcohol harm reduction advertisements: a content analysis of topic, objective, emotional tone, execution and target audience . BMC Public Health , 17 , 13 .

Eckler P. , Bolls P. ( 2011 ) Spreading the virus: emotional tone of viral advertising and its effect on forwarding intentions and attitudes . Journal of Interactive Advertising , 11 , 1 – 11 .

Effective Public Health Practice Project . ( 2019 ) Quality assessment tool for quantitative studies. http://www.ephpp.ca/tools.html .

Elder R. W. , Shults R. A. , Sleet D. A. , Nichols J. L. , Thompson R. S. , Rajab W , et al.  ( 2004 ) Effectiveness of mass media campaigns for reducing drinking and driving and alcohol-involved crashes: a systematic review . American Journal of Preventive Medicine , 27 , 57 – 65 .

Esrick J. , Kagan R. G. , Carnevale J. T. , Valenti M. , Rots G. , Dash K. ( 2019 ) Can scare tactics and fear-based messages help deter substance misuse: a systematic review of recent (2005–2017) research, Drugs-Education Prevention and Policy , 26 , 209 – 218 .

Estrada Y. , Lee T. K. , Huang S. , Tapia M. I. , Velazquez M. R. , Martinez M. J. et al.  ( 2017 ) Parent-centered prevention of risky behaviors among hispanic youths in Florida . American Journal of Public Health , 107 , 607 – 613 .

Faseur T. , Geuens M. ( 2010 ) Communicating the right emotion to generate help for connected versus unconnected others . Communication Research , 37 , 498 – 521 .

Hendriks H. , van den Putte B. , de Bruijn G. J. ( 2014 ) Changing the conversation: the influence of emotions on conversational valence and alcohol consumption . Prevention Science , 15 , 684 – 693 .

Hornik J. , Ofir C. , Rachamim M. ( 2016 ) Quantitative evaluation of persuasive appeals using comparative meta-analysis . The Communication Review , 19 , 192 – 222 .

Hornik J. , Ofir C. , Rachamim M. ( 2017 ) Advertising appeals, moderators, and impact on persuasion: a quantitative assessment creates a hierarchy of appeals . Journal of Advertising Research , 57 , 305 – 318 .

Jackson N. , Waters E ; Promotion Guidelines for Systematic Reviews in Health and Taskforce Public Health . ( 2005 ) Criteria for the systematic review of health promotion and public health interventions . Health Promotion International , 20 , 367 – 374 .

Jäger T. , Eisend M. ( 2013 ) Effects of fear-arousing and humorous appeals in social marketing advertising: the moderating role of prior attitude toward the advertised behavior . Journal of Current Issues & Research in Advertising , 34 , 125 – 134 .

Jenkin G. , Madhvani N. , Signal L. , Bowers S. ( 2014 ) A systematic review of persuasive marketing techniques to promote food to children on television . Obesity Reviews , 15 , 281 – 293 .

Jordan A. , Bleakley A. , Hennessy M. , Vaala S. , Glanz K. , Strasser A. A. ( 2015 ) Sugar-sweetened beverage-related public service advertisements and their influence on parents . American Behavioral Scientist , 59 , 1847 – 1865 .

Kaye S. A. , Lewis I. , Algie J. , White M. J. ( 2016 ) Young drivers’ responses to anti-speeding advertisements: comparison of self-report and objective measures of persuasive processing and outcomes . Traffic Injury Prevention , 17 , 352 – 358 .

Kemp E. , Kennett-Hensel P. A. , Kees J. ( 2013 ) Pulling on the heartstrings: examining the effects of emotions and gender in persuasive appeals . Journal of Advertising , 42 , 69 – 79 .

Lau-Gesk L. , Meyers-Levy J. ( 2009 ) Emotional persuasion: when the valence versus the resource demands of emotions influence consumers’ attitudes . Journal of Consumer Research , 36 , 585 – 599 .

Lee M. J. ( 2018 ) College students’ responses to emotional anti-alcohol abuse media messages: should we scare or amuse them? Health Promotion Practice , 19 , 465 – 474 .

Lee M. J. , Ferguson M. A. ( 2002 ) Effects of anti-tobacco advertisements based on risk-taking tendencies: realistic fear vs. vulgar humor . Journalism & Mass Communication Quarterly , 79 , 945 – 963 .

Lennon R. , Rentfro R. , Bay O. ( 2010 ) Social marketing and distracted driving behaviors among young adults: the effectiveness of fear appeals . Academy of Marketing Studies Journal , 14 , 95 – 113 .

Lerner J. S. , Keltner D. ( 2000 ) Beyond valence: Toward a model of emotion-specific influences on judgement and choice . Cognition & Emotion , 14 , 473 – 493 .

Lewis I. , Watson B. , White K. M. ( 2008 ) An examination of message-relevant affect in road safety messages: should road safety advertisements aim to make us feel good or bad? Transportation Research Part F: Traffic Psychology and Behaviour , 11 , 403 – 417 .

Matthes J. , Wonneberger A. ( 2014 ) The skeptical green consumer revisited: testing the relationship between green consumerism and skepticism toward advertising . Journal of Advertising , 43 , 115 – 127 .

Mattila A. S. ( 1999 ) Do emotional appeals work for services? International Journal of Service Industry Management , 10 , 292 – 306 .

Moher D. , Liberati A. , Tetzlaff J. , Altman D. G. ; PRISMA Group . ( 2009 ) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Annals of Internal Medicine , 151 , 264 – 269 .

Moran N. , Bagchi R. ( 2019 ) The power of emotional benefits: examining the role of benefit focus on donation behavior . Journal of Advertising , 48 , 284 – 291 .

Mukherjee A. , Dubé L. ( 2012 ) Mixing emotions: the use of humor in fear advertising . Journal of Consumer Behaviour , 11 , 147 – 161 .

Nabi R. L. ( 2015 ) Emotional flow in persuasive health messages . Health Communication , 30 , 114 – 124 .

Noble G. , Pomering A. , W. Johnson L. ( 2014 ) Gender and message appeal: their influence in a pro-environmental social advertising context . Journal of Social Marketing , 4 , 4 – 21 .

O’Keefe D. J. , Jensen J. D. ( 2009 ) The relative persuasiveness of gain-framed and loss-framed messages for encouraging disease detection behaviors: a meta-analytic review . Journal of Communication , 59 , 296 – 316 .

Passyn K. , Sujan M. ( 2006 ) Self accountability emotions and fear appeals: motivating behavior . Journal of Consumer Research , 32 , 583 – 589 .

Pitt L. F. , Berthon P. , Caruana A. , Berthon J.-P. ( 2005 ) The state of theory in three premier advertising journals: a research note . International Journal of Advertising , 24 , 241 – 249 .

Plant B. , Reeza F. , Irwin J. D. ( 2011 ) A systematic review of how anti-speeding advertisements are evaluated . Journal of the Australasian College of Road Safety , 22 , 18 .

Plant B. R. C. , Irwin J. D. , Chekaluk E. ( 2017 ) The effects of anti-speeding advertisements on the simulated driving behaviour of young drivers . Accident; Analysis and Prevention , 100 , 65 – 74 .

Previte J. , Russell-Bennett R. , Parkinson J. ( 2015 ) Shaping safe drinking cultures: evoking positive emotion to promote moderate-drinking behaviour . International Journal of Consumer Studies , 39 , 12 – 24 .

Rodrigue J. R. , Fleishman A. , Vishnevsky T. , Fitzpatrick S. , Boger M. ( 2014 ) Organ donation video messaging: differential appeal, emotional valence, and behavioral intention . Clinical Transplantation , 28 , 1184 – 1192 .

Sarstedt M. , Bengart P. , Shaltoni A. M. , Lehmann S. ( 2018 ) The use of sampling methods in advertising research: a gap between theory and practice . International Journal of Advertising , 37 , 650 – 663 .

Scherer K. R. ( 2005 ) What are emotions? And how can they be measured? Social Science Information , 44 , 695 – 729 .

Skurka C. , Niederdeppe J. , Romero-Canyas R. , Acup D. ( 2018 ) Pathways of influence in emotional appeals: benefits and tradeoffs of using fear or humor to promote climate change-related intentions and risk perceptions . Journal of Communication , 68 , 169 – 193 .

Small D. A. , Verrochi N. M. ( 2009 ) The face of need: facial emotion expression on charity advertisements . Journal of Marketing Research , 46 , 777 – 787 .

Struckman‐Johnson C. , Struckman‐Johnson D. , Gilliland R. C. , Ausman A. ( 1994 ) Effect of persuasive appeals in AIDS PSAs and condom commercials on intentions to use condoms 1, Journal of Applied Social Psychology , 24 , 2223 – 2244 .

Sun H. J. ( 2015 ) A study on the development of public campaign messages for organ donation promotion in Korea . Health Promotion International , 30 , 903 – 918 .

Tannenbaum M. B. , Hepler J. , Zimmerman R. S. , Saul L. , Jacobs S. , Wilson K. et al.  ( 2015 ) Appealing to fear: a meta-analysis of fear appeal effectiveness and theories . Psychological Bulletin , 141 , 1178 – 1204 .

Taute H. A. , McQuitty S. , Sautter E. P. ( 2011 ) Emotional information management and responses to emotional appeals . Journal of Advertising , 40 , 31 – 43 .

Tay R. ( 2005 ) The effectiveness of enforcement and publicity campaigns on serious crashes involving young male drivers: are drink driving and speeding similar? Accident; Analysis and Prevention , 37 , 922 – 929 .

Tay R. ( 2011 ) Drivers’ perception of two seatbelt wearing advertisements with different emotional appeals and cultural settings . Journal of the Australasian College of Road Safety , 22 , 82 .

Thainiyom P. , Elder K. ( 2017 ) Emotional Appeals in HIV Prevention Campaigns: unintended stigma effects . American Journal of Health Behavior , 41 , 390 – 400 .

Thomas B. H. , Ciliska D. , Dobbins M. , Micucci S. ( 2004 ) A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions . Worldviews on Evidence-Based Nursing , 1 , 176 – 184 .

Vaala S. E. , Bleakley A. , Hennessy M. , Jordan A. B. ( 2016 ) Weight stigmatization moderates the effects of sugar-sweetened beverage-related PSAs among US parents . Media Psychology , 19 , 534 – 560 .

Wadsworth D. D. , Hallam J. S. ( 2010 ) Effect of a web site intervention on physical activity of college females . American Journal of Health Behavior , 34 , 60 – 69 .

Wang J. M. , Bao J. , Wang C. C. , Wu L. C. ( 2017 ) The impact of different emotional appeals on the purchase intention for green products: The moderating effects of green involvement and Confucian cultures . Sustainable Cities and Society , 34 , 32 – 42 .

Williams C. L. , Grechanaia T. , Romanova O. , Komro K. A. , Perry C. L. , Farbakhsh K. ( 2001 ) Russian-American partners for prevention: adaptation of a school-based parent-child programme for alcohol use prevention . European Journal of Public Health , 11 , 314 – 321 .

Williams J. D. , Lee W.-N. , Haugtvedt C. P. ( 2004 ) Diversity in Advertising: Broadening the Scope of Research Directions . Psychology Press, Hove, East Sussex, United Kingdom .

Willmott T. , Pang B. , Rundle-Thiele S. , Badejo A. ( 2019 ) Reported theory use in electronic health weight management interventions targeting young adults: a systematic review . Health Psychology Review , 13 , 295 – 317 .

Witte K. , Allen M. ( 2000 ) A meta-analysis of fear appeals: implications for effective public health campaigns . Health Education & Behavior , 27 , 591 – 615 .

Wu C. H. , Sundiman D. , Kao S. C. , Chen C. H. ( 2018 ) Emotion induction in click intention of picture advertisement: a field examination . Journal of Internet Commerce , 17 , 356 – 382 .

Yoon H. J. ( 2018 ) Using humour to increase effectiveness of shameful health issue advertising: testing the effects of health worry level . International Journal of Advertising , 37 , 914 – 936 .

Yoon H. J. , Tinkham S. F. ( 2013 ) Humorous threat persuasion in advertising: The effects of humor, threat intensity, and issue involvement . Journal of Advertising , 42 , 30 – 41 .

Yousef M. , Dietrich T. , Rundle-Thiele S. ( 2021 ) Social advertising effectiveness in driving action: a study of positive, negative and coactive appeals on social media . International Journal of Environmental Research and Public Health , 18 , 5954 .

Yousef M. , Dietrich T. , Torrisi G. ( 2021 ) Positive, negative or both? Assessing emotional appeals effectiveness in anti-drink driving advertisements . Social Marketing Quarterly , 27 , 195 – 212 .

Zemack-Rugar Y. , Klucarova-Travani S. ( 2018 ) Should donation ads include happy victim images? The moderating role of regulatory focus . Marketing Letters , 29 , 421 – 434 .

Supplementary data

Email alerts, citing articles via.

• Recommend to Your Librarian
• Journals Career Network

Affiliations

• Online ISSN 1460-2245
• Print ISSN 0957-4824
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Systematic Review
• Open access
• Published: 23 May 2024

Systematic literature review of real-world evidence for treatments in HR+/HER2- second-line LABC/mBC after first-line treatment with CDK4/6i

• Veronique Lambert   ORCID: orcid.org/0000-0002-6984-0038 1 ,
• Sarah Kane   ORCID: orcid.org/0009-0006-9341-4836 2   na1 ,
• Belal Howidi   ORCID: orcid.org/0000-0002-1166-7631 2   na1 ,
• Bao-Ngoc Nguyen   ORCID: orcid.org/0000-0001-6026-2270 2   na1 ,
• David Chandiwana   ORCID: orcid.org/0009-0002-3499-2565 3 ,
• Yan Wu   ORCID: orcid.org/0009-0008-3348-9232 1 ,
• Michelle Edwards   ORCID: orcid.org/0009-0001-4292-3140 3 &
• Imtiaz A. Samjoo   ORCID: orcid.org/0000-0003-1415-8055 2   na1  

BMC Cancer volume  24 , Article number:  631 ( 2024 ) Cite this article

1 Altmetric

Metrics details

Cyclin-dependent kinase 4 and 6 inhibitors (CDK4/6i) combined with endocrine therapy (ET) are currently recommended by the National Comprehensive Cancer Network (NCCN) guidelines and the European Society for Medical Oncology (ESMO) guidelines as the first-line (1 L) treatment for patients with hormone receptor-positive, human epidermal growth factor receptor 2-negative, locally advanced/metastatic breast cancer (HR+/HER2- LABC/mBC). Although there are many treatment options, there is no clear standard of care for patients following 1 L CDK4/6i. Understanding the real-world effectiveness of subsequent therapies may help to identify an unmet need in this patient population. This systematic literature review qualitatively synthesized effectiveness and safety outcomes for treatments received in the real-world setting after 1 L CDK4/6i therapy in patients with HR+/ HER2- LABC/mBC.

MEDLINE®, Embase, and Cochrane were searched using the Ovid® platform for real-world evidence studies published between 2015 and 2022. Grey literature was searched to identify relevant conference abstracts published from 2019 to 2022. The review was conducted in accordance with PRISMA guidelines (PROSPERO registration: CRD42023383914). Data were qualitatively synthesized and weighted average median real-world progression-free survival (rwPFS) was calculated for NCCN/ESMO-recommended post-1 L CDK4/6i treatment regimens.

Twenty records (9 full-text articles and 11 conference abstracts) encompassing 18 unique studies met the eligibility criteria and reported outcomes for second-line (2 L) treatments after 1 L CDK4/6i; no studies reported disaggregated outcomes in the third-line setting or beyond. Sixteen studies included NCCN/ESMO guideline-recommended treatments with the majority evaluating endocrine-based therapy; five studies on single-agent ET, six studies on mammalian target of rapamycin inhibitors (mTORi) ± ET, and three studies with a mix of ET and/or mTORi. Chemotherapy outcomes were reported in 11 studies. The most assessed outcome was median rwPFS; the weighted average median rwPFS was calculated as 3.9 months (3.3-6.0 months) for single-agent ET, 3.6 months (2.5–4.9 months) for mTORi ± ET, 3.7 months for a mix of ET and/or mTORi (3.0–4.0 months), and 6.1 months (3.7–9.7 months) for chemotherapy. Very few studies reported other effectiveness outcomes and only two studies reported safety outcomes. Most studies had heterogeneity in patient- and disease-related characteristics.

Conclusions

The real-world effectiveness of current 2 L treatments post-1 L CDK4/6i are suboptimal, highlighting an unmet need for this patient population.

Peer Review reports

Introduction

Breast cancer (BC) is the most diagnosed form of cancer in women with an estimated 2.3 million new cases diagnosed worldwide each year [ 1 ]. BC is the second leading cause of cancer death, accounting for 685,000 deaths worldwide per year [ 2 ]. By 2040, the global burden associated with BC is expected to surpass three million new cases and one million deaths annually (due to population growth and aging) [ 3 ]. Numerous factors contribute to global disparities in BC-related mortality rates, including delayed diagnosis, resulting in a high number of BC cases that have progressed to locally advanced BC (LABC) or metastatic BC (mBC) [ 4 , 5 , 6 ]. In the United States (US), the five-year survival rate for patients who progress to mBC is three times lower (31%) than the overall five-year survival rate for all stages (91%) [ 6 , 7 ].

Hormone receptor (HR) positive (i.e., estrogen receptor and/or progesterone receptor positive) coupled with negative human epidermal growth factor 2 (HER2) expression is the most common subtype of BC, accounting for ∼ 60–70% of all BC cases [ 8 , 9 ]. Historically, endocrine therapy (ET) through estrogen receptor modulation and/or estrogen deprivation has been the standard of care for first-line (1 L) treatment of HR-positive/HER2-negative (HR+/HER2-) mBC [ 10 ]. However, with the approval of the cyclin-dependent kinase 4/6 inhibitor (CDK4/6i) palbociclib in combination with the aromatase inhibitor (AI) letrozole in 2015 by the US Food and Drug Administration (FDA), 1 L treatment practice patterns have evolved such that CDK4/6i (either in combination with AIs or with fulvestrant) are currently considered the standard of care [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ]. Other CDK4/6i (ribociclib and abemaciclib) in combination with ET are approved for the treatment of HR+/HER2- LABC/mBC; 1 L use of ribociclib in combination with an AI was granted FDA approval in March 2017 for postmenopausal women (with expanded approval in July 2018 for pre/perimenopausal women and for use in 1 L with fulvestrant for patients with disease progression on ET as well as for postmenopausal women), and abemaciclib in combination with fulvestrant was granted FDA approval in September 2017 for patients with disease progression following ET and as monotherapy in cases where disease progression occurs following ET and prior chemotherapy in mBC (with expanded approval in February 2018 for use in 1 L in combination with an AI for postmenopausal women) [ 18 , 19 , 20 , 21 ].

Clinical trials investigating the addition of CDK4/6i to ET have demonstrated significant improvement in progression-free survival (PFS) and significant (ribociclib) or numerical (palbociclib and abemaciclib) improvement in overall survival (OS) compared to ET alone in patients with HR+/HER2- advanced or mBC, making this combination treatment the recommended option in the 1 L setting [ 22 , 23 , 24 , 25 , 26 , 27 ]. However, disease progression occurs in a significant portion of patients after 1 L CDK4/6i treatment [ 28 ] and the optimal treatment sequence after progression on CDK4/6i remains unclear [ 29 ]. At the time of this review (literature search conducted December 14, 2022), guidelines by the National Comprehensive Cancer Network (NCCN) and the European Society for Medical Oncology (ESMO) recommend various options for the treatment of HR+/HER2- advanced BC in the second-line (2 L) setting, including fulvestrant monotherapy, mammalian target of rapamycin inhibitors (mTORi; e.g., everolimus) ± ET, alpelisib + fulvestrant (if phosphatidylinositol-4,5-bisphosphate 3-kinase catalytic subunit alpha mutation positive [PIK3CA-m+]), poly-ADP ribose polymerase inhibitors (PARPi) including olaparib or talazoparib (if breast cancer gene/partner and localizer of BRCA2 positive [BRCA/PALB2m+]), and chemotherapy (in cases when a visceral crisis is present) [ 15 , 16 ]. CDK4/6i can also be used in 2 L [ 16 , 30 ]; however, limited data are available to support CDK4/6i rechallenge after its use in the 1 L setting [ 15 ]. Depending on treatments used in the 1 L and 2 L settings, treatment in the third-line setting is individualized based on the patient’s response to prior treatments, tumor load, duration of response, and patient preference [ 9 , 15 ]. Understanding subsequent treatments after 1 L CDK4/6i, and their associated effectiveness, is an important focus in BC research.

Treatment options for HR+/HER2- LABC/mBC continue to evolve, with ongoing research in both clinical trials and in the real-world setting. Real-world evidence (RWE) offers important insights into novel therapeutic regimens and the effectiveness of treatments for HR+/HER2- LABC/mBC. The effectiveness of the current treatment options following 1 L CDK4/6i therapy in the real-world setting highlights the unmet need in this patient population and may help to drive further research and drug development. In this study, we conducted a systematic literature review (SLR) to qualitatively summarize the effectiveness and safety of treatment regimens in the real-world setting after 1 L treatment with CDK4/6i in patients with HR+/HER2- LABC/mBC.

Literature search

An SLR was performed in accordance with the Cochrane Handbook for Systematic Reviews of Interventions [ 31 ] and reported in alignment with the Preferred Reporting Items for Systematic Literature Reviews and Meta-Analyses (PRISMA) statement [ 32 ] to identify all RWE studies assessing the effectiveness and safety of treatments used for patients with HR+/HER2- LABC/mBC following 1 L CDK4/6i therapy and received subsequent treatment in 2 L and beyond (2 L+). The Ovid® platform was used to search MEDLINE® (including Epub Ahead of Print and In-Process, In-Data-Review & Other Non-Indexed Citations), Ovid MEDLINE® Daily, Embase, Cochrane Central Register of Controlled Trials, and Cochrane Database of Systematic Reviews by an experienced medical information specialist. The MEDLINE® search strategy was peer-reviewed independently by a senior medical information specialist before execution using the Peer Review of Electronic Search Strategies (PRESS) checklist [ 33 ]. Searches were conducted on December 14, 2022. The review protocol was developed a priori and registered with the International Prospective Register of Systematic Review (PROSPERO; CRD42023383914) which outlined the population, intervention, comparator, outcome, and study design (PICOS) criteria and methodology used to conduct the review (Table  1 ).

Search strategies utilized a combination of controlled vocabulary (e.g., “HER2 Breast Cancer” or “HR Breast Cancer”) and keywords (e.g., “Retrospective studies”). Vocabulary and syntax were adjusted across databases. Published and validated filters were used to select for study design and were supplemented using additional medical subject headings (MeSH) terms and keywords to select for RWE and nonrandomized studies [ 34 ]. No language restrictions were included in the search strategy. Animal-only and opinion pieces were removed from the results. The search was limited to studies published between January 2015 and December 2022 to reflect the time at which FDA approval was granted for the first CDK4/6i agent (palbociclib) in combination with AI for the treatment of LABC/mBC [ 35 ]. Further search details are presented in Supplementary Material 1 .

Grey literature sources were also searched to identify relevant abstracts and posters published from January 2019 to December 2022 for prespecified relevant conferences including ESMO, San Antonio Breast Cancer Symposium (SABCS), American Society of Clinical Oncology (ASCO), the International Society for Pharmacoeconomics and Outcomes Research (ISPOR US), and the American Association for Cancer Research (AACR). A search of ClinicalTrials.gov was conducted to validate the findings from the database and grey literature searches.

Study selection, data extraction & weighted average calculation

Studies were screened for inclusion using DistillerSR Version 2.35 and 2.41 (DistillerSR Inc. 2021, Ottawa, Canada) by two independent reviewers based on the prespecified PICOS criteria (Table  1 ). A third reviewer was consulted to resolve any discrepancies during the screening process. Studies were included if they reported RWE on patients aged ≥ 18 years with HR+/HER2- LABC/mBC who received 1 L CDK4/6i treatment and received subsequent treatment in 2 L+. Studies were excluded if they reported the results of clinical trials (i.e., non-RWE), were published in any language other than English, and/or were published prior to 2015 (or prior to 2019 for conference abstracts and posters). For studies that met the eligibility criteria, data relating to study design and methodology, details of interventions, patient eligibility criteria and baseline characteristics, and outcome measures such as efficacy, safety, tolerability, and patient-reported outcomes (PROs), were extracted (as available) using a Microsoft Excel®-based data extraction form (Microsoft Corporation, WA, USA). Data extraction was performed by a single reviewer and was confirmed by a second reviewer. Multiple publications identified for the same RWE study, patient population, and setting that reported data for the same intervention were linked and extracted as a single publication. Weighted average median real-world progression-free survival (rwPFS) values were calculated by considering the contribution to the median rwPFS of each study proportional to its respective sample size. These weighted values were then used to compute the overall median rwPFS estimate.

Quality assessment

The Newcastle-Ottawa scale (NOS) for nonrandomized (cohort) studies was used to assess the risk of bias for published, full-text studies [ 36 ]. The NOS allocates a maximum of nine points for the least risk of bias across three domains: (1) Formation of study groups (four points), (2) Comparability between study groups (two points), (3) Outcome ascertainment (three points). NOS scores can be categorized in three groups: very high risk of bias (0 to 3 points), high risk of bias (4 to 6), and low risk of bias (7 to 9) [ 37 ]. Risk of bias assessment was performed by one reviewer and validated by a second independent reviewer to verify accuracy. Due to limited methodological data by which to assess study quality, risk of bias assessment was not performed on conference abstracts or posters. An amendment to the PROSPERO record (CRD42023383914) for this study was submitted in relation to the quality assessment method (specifying usage of the NOS).

The database search identified 3,377 records; after removal of duplicates, 2,759 were screened at the title and abstract stage of which 2,553 were excluded. Out of the 206 reports retrieved and assessed for eligibility, an additional 187 records were excluded after full-text review; most of these studies were excluded for having patients with mixed lines of CDK4/6i treatment (i.e., did not receive CDK4/6i exclusively in 1 L) (Fig.  1 and Table S1 ). The grey literature search identified 753 records which were assessed for eligibility; of which 752 were excluded mainly due to the population not meeting the eligibility criteria (Fig.  1 ). In total, the literature searches identified 20 records (9 published full-text articles and 11 conference abstracts/posters) representing 18 unique RWE studies that met the inclusion criteria. The NOS quality scores for the included full-text articles are provided in Table S2 . The scores ranged from four to six points (out of a total score of nine) and the median score was five, indicating that all the studies suffered from a high risk of bias [ 37 ].

Most studies were retrospective analyses of chart reviews or medical registries, and all studies were published between 2017 and 2022 (Table S3 ). Nearly half of the RWE studies (8 out of 18 studies) were conducted in the US [ 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ], while the remaining studies included sites in Canada, China, Germany, Italy, Japan, and the United Kingdom [ 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 ]. Sample sizes ranged from as few as 4 to as many as 839 patients across included studies, with patient age ranging from 26 to 86 years old.

Although treatment characteristics in the 1 L setting were not the focus of the present review, these details are captured in Table S3 . Briefly, several RWE studies reported 1 L CDK4/6i use in combination with ET (8 out of 18 studies) or as monotherapy (2 out of 18 studies) (Table S3 ). Treatments used in combination with 1 L CDK4/6i included letrozole, fulvestrant, exemestane, and anastrozole. Where reported (4 out of 18 studies), palbociclib was the most common 1 L CDK4/6i treatment. Many studies (8 out of 18 studies) did not report which specific CDK4/6i treatment(s) were used in 1 L or if its administration was in combination or monotherapy.

Characteristics of treatments after 1 L CDK4/6i therapy

Across all studies included in this review, effectiveness and safety data were only available for treatments administered in the 2 L setting after 1 L CDK4/6i treatment. No studies were identified that reported outcomes for patients treated in the third-line setting or beyond after 1 L CDK4/6i treatment. All 18 studies reported effectiveness outcomes in 2 L, with only two of these studies also describing 2 L safety outcomes. The distribution of outcomes reported in these studies is provided in Table S4 . Studies varied in their reporting of outcomes for 2 L treatments; some studies reported outcomes for a group of 2 L treatments while others described independent outcomes for specific 2 L treatments (i.e., everolimus, fulvestrant, or chemotherapy agents such as eribulin mesylate) [ 42 , 45 , 50 , 54 , 55 ]. Due to the heterogeneity in treatment classes reported in these studies, this data was categorized (as described below) to align with the guidelines provided by NCCN and ESMO [ 15 , 16 ]. The treatment class categorizations for the purpose of this review are: single-agent ET (patients who exclusively received a single-agent ET after 1 L CDK4/6i treatment), mTORi ± ET (patients who exclusively received an mTORi with or without ET after 1 L CDK4/6i treatment), mix of ET and/or mTORi (patients who may have received only ET, only mTORi, and/or both treatments but the studies in this group lacked sufficient information to categorize these patients in the “single-agent ET” or “mTOR ± ET” categories), and chemotherapy (patients who exclusively received chemotherapy after 1 L CDK4/6i treatment). Despite ESMO and NCCN guidelines indicating that limited evidence exists to support rechallenge with CDK4/6i after 1 L CDK4/6i treatment [ 15 , 16 ], two studies reported outcomes for this treatment approach. Data for such patients were categorized as “ CDK4/6i ± ET ” as it was unclear how many patients receiving CDK4/6i rechallenge received concurrent ET. All other patient groups that lacked sufficient information or did not report outcome/safety data independently (i.e., grouped patients with mixed treatments) to categorize as one of the treatment classes described above were grouped as “ other ”.

The majority of studies reported effectiveness outcomes for endocrine-based therapy after 1 L CDK4/6i treatment; five studies for single-agent ET, six studies for mTORi ± ET, and three studies for a mix of ET and/or mTORi (Fig.  2 ). Eleven studies reported effectiveness outcomes for chemotherapy after 1 L CDK4/6i treatment, and only two studies reported effectiveness outcomes for CDK4/6i rechallenge ± ET. Eight studies that described effectiveness outcomes were grouped into the “other” category. Safety data was only reported in two studies: one study evaluating the chemotherapy agent eribulin mesylate and one evaluating the mTORi everolimus.

Effectiveness outcomes

Real-world progression-free survival

Median rwPFS was described in 13 studies (Tables  2 and Table S5 ). Across the 13 studies, the median rwPFS ranged from 2.5 months [ 49 ] to 17.3 months [ 39 ]. Out of the 13 studies reporting median rwPFS, 10 studies reported median rwPFS for a 2 L treatment recommended by ESMO and NCCN guidelines, which ranged from 2.5 months [ 49 ] to 9.7 months [ 45 ].

Weighted average median rwPFS was calculated for 2 L treatments recommended by both ESMO and NCCN guidelines (Fig.  3 ). The weighted average median rwPFS for single-agent ET was 3.9 months ( n  = 92 total patients) and was derived using data from two studies reporting median rwPFS values of 3.3 months ( n  = 70) [ 38 ] and 6.0 months ( n  = 22) [ 40 ]. For one study ( n  = 7) that reported outcomes for single agent ET, median rwPFS was not reached during the follow-up period; as such, this study was excluded from the weighted average median rwPFS calculation [ 49 ].

The weighted average median rwPFS for mTORi ± ET was 3.6 months ( n  = 128 total patients) and was derived based on data from 3 studies with median rwPFS ranging from 2.5 months ( n  = 4) [ 49 ] to 4.9 months ( n  = 25) [ 54 ] (Fig.  3 ). For patients who received a mix of ET and/or mTORi but could not be classified into the single-agent ET or mTORi ± ET treatment classes, the weighted average median rwPFS was calculated to be 3.7 months ( n  = 17 total patients). This was calculated based on data from two studies reporting median rwPFS values of 3.0 months ( n  = 5) [ 46 ] and 4.0 months ( n  = 12) [ 49 ]. Notably, one study of patients receiving ET and/or everolimus reported a median rwPFS duration of 3.0 months; however, this study was excluded from the weighted average median rwPFS calculation for the ET and/or mTORi class as the sample size was not reported [ 53 ].

The weighted average median rwPFS for chemotherapy was 6.1 months ( n  = 499 total patients), calculated using data from 7 studies reporting median rwPFS values ranging from 3.7 months ( n  = 249) [ 38 ] to 9.7 months ( n  = 121) [ 45 ] (Fig.  3 ). One study with a median rwPFS duration of 5.6 months was not included in the weighted average median rwPFS calculation as the study did not report the sample size [ 53 ]. A second study was excluded from the calculation since the reported median rwPFS was not reached during the study period ( n  = 7) [ 41 ].

Although 2 L CDK4/6i ± ET rechallenge lacks sufficient information to support recommendation by ESMO and NCCN guidelines, the limited data currently available for this treatment have shown promising results. Briefly, two studies reported median rwPFS for CDK4/6i ± ET with values of 8.3 months ( n  = 302) [ 38 ] and 17.3 months ( n  = 165) (Table  2 ) [ 39 ]. The remaining median rwPFS studies reported data for patients classified as “Other” (Table S5 ). The “Other” category included median rwPFS outcomes from seven studies, and included a myriad of treatments (e.g., ET, mTOR + ET, chemotherapy, CDK4/6i + ET, alpelisib + fulvestrant, chidamide + ET) for which disaggregated median rwPFS values were not reported.

Overall survival

Median OS for 2 L treatment was reported in only three studies (Table  2 ) [ 38 , 42 , 43 ]. Across the three studies, the 2 L median OS ranged from 5.2 months ( n  = 3) [ 43 ] to 35.7 months ( n  = 302) [ 38 ]. Due to the lack of OS data in most of the studies, weighted averages could not be calculated. No median OS data was reported for the single-agent ET treatment class whereas two studies reported median OS for the mTORi ± ET treatment class, ranging from 5.2 months ( n  = 3) [ 43 ] to 21.8 months ( n  = 54) [ 42 ]. One study reported 2 L median OS of 24.8 months for a single patient treated with chemotherapy [ 43 ]. The median OS data in the CDK4/6i ± ET rechallenge group was 35.7 months ( n  = 302) [ 38 ].

Patient mortality was reported in three studies [ 43 , 44 , 45 ]. No studies reported mortality for the single-agent ET treatment class and only one study reported this outcome for the mTORi ± ET treatment class, where 100% of patients died ( n  = 3) as a result of rapid disease progression [ 43 ]. For the chemotherapy class, one study reported mortality for one patient receiving 2 L capecitabine [ 43 ]. An additional study reported eight deaths (21.7%) following 1 L CDK4/6i treatment; however, this study did not disclose the 2 L treatments administered to these patients [ 44 ].

Other clinical endpoints

The studies included limited information on additional clinical endpoints; two studies reported on time-to-discontinuation (TTD), two reported on duration of response (DOR), and one each on time-to-next-treatment (TTNT), time-to-progression (TTP), objective response rate (ORR), clinical benefit rate (CBR), and stable disease (Tables  2 and Table S5 ).

Safety, tolerability, and patient-reported outcomes

Safety and tolerability data were reported in two studies [ 40 , 45 ]. One study investigating 2 L administration of the chemotherapy agent eribulin mesylate reported 27 patients (22.3%) with neutropenia, 3 patients (2.5%) with febrile neutropenia, 10 patients (8.3%) with peripheral neuropathy, and 14 patients (11.6%) with diarrhea [ 45 ]. Of these, neutropenia of grade 3–4 severity occurred in 9 patients (33.3%) [ 45 ]. A total of 55 patients (45.5%) discontinued eribulin mesylate treatment; 1 patient (0.83%) discontinued treatment due to adverse events [ 45 ]. Another study reported that 5 out of the 22 patients receiving the mTORi everolimus combined with ET in 2 L (22.7%) discontinued treatment due to toxicity [ 40 ]. PROs were not reported in any of the studies included in the SLR.

The objective of this study was to summarize the existing RWE on the effectiveness and safety of therapies for patients with HR+/HER2- LABC/mBC after 1 L CDK4/6i treatment. We identified 18 unique studies reporting specifically on 2 L treatment regimens after 1 L CDK4/6i treatment. The weighted average median rwPFS for NCCN- and ESMO- guideline recommended 2 L treatments ranged from 3.6 to 3.9 months for ET-based treatments and was 6.1 months when including chemotherapy-based regimens. Treatment selection following 1 L CDK4/6i therapy remains challenging primarily due to the suboptimal effectiveness or significant toxicities (e.g., chemotherapy) associated with currently available options [ 56 ]. These results highlight that currently available 2 L treatments for patients with HR+/HER2- LABC/mBC who have received 1 L CDK4/6i are suboptimal, as evidenced by the brief median rwPFS duration associated with ET-based treatments, or notable side effects and toxicity linked to chemotherapy. This conclusion is aligned with a recent review highlighting the limited effectiveness of treatment options for HR+/HER2- LABC/mBC patients post-CDK4/6i treatment [ 56 , 57 ]. Registrational trials which have also shed light on the short median PFS of 2–3 months achieved by ET (i.e., fulvestrant) after 1 L CDK4/6i therapy emphasize the need to develop improved treatment strategies aimed at prolonging the duration of effective ET-based treatment [ 56 ].

The results of this review reveal a paucity of additional real-world effectiveness and safety evidence after 1 L CDK4/6i treatment in HR+/HER2- LABC/mBC. OS and DOR were only reported in two studies while other clinical endpoints (i.e., TTD, TTNT, TTP, ORR, CBR, and stable disease) were only reported in one study each. Similarly, safety and tolerability data were only reported in two studies each, and PROs were not reported in any study. This hindered our ability to provide a comprehensive assessment of real-world treatment effectiveness and safety following 1 L CDK4/6i treatment. The limited evidence may be due to the relatively short period of time that has elapsed since CDK4/6i first received US FDA approval for 1 L treatment of HR+/HER2- LABC/mBC (2015) [ 35 ]. As such, almost half of our evidence was informed by conference abstracts. Similarly, no real-world studies were identified in our review that reported outcomes for treatments in the third- or later-lines of therapy after 1 L CDK4/6i treatment. The lack of data in this patient population highlights a significant gap which limits our understanding of the effectiveness and safety for patients receiving later lines of therapy. As more patients receive CDK4/6i therapy in the 1 L setting, the number of patients requiring subsequent lines of therapy will continue to grow. Addressing this data gap over time will be critical to improve outcomes for patients with HR+/HER2- LABC/mBC following 1 L CDK4/6i therapy.

There are several strengths of this study, including adherence to the guidelines outlined in the Cochrane Handbook to ensure a standardized and reliable approach to the SLR [ 58 ] and reporting of the SLR following PRISMA guidelines to ensure transparency and reproducibility [ 59 ]. Furthermore, the inclusion of only RWE studies allowed us to assess the effectiveness of current standard of care treatments outside of a controlled environment and enabled us to identify an unmet need in this patient population.

This study had some notable limitations, including the lack of safety and additional effectiveness outcomes reported. In addition, the dearth of studies reporting PROs is a limitation, as PROs provide valuable insight into the patient experience and are an important aspect of assessing the impact of 2 L treatments on patients’ quality of life. The studies included in this review also lacked consistent reporting of clinical characteristics (e.g., menopausal status, sites of metastasis, prior surgery) making it challenging to draw comprehensive conclusions or comparisons based on these factors across the studies. Taken together, there exists an important gap in our understanding of the long-term management of patients with HR+/HER2- LABC/mBC. Additionally, the effectiveness results reported in our evidence base were informed by small sample sizes; many of the included studies reported median rwPFS based on less than 30 patients [ 39 , 40 , 41 , 46 , 49 , 51 , 60 ], with two studies not reporting the sample size at all [ 47 , 53 ]. This may impact the generalizability and robustness of the results. Relatedly, the SLR database search was conducted in December 2022; as such, novel agents (e.g., elacestrant and capivasertib + fulvestrant) that have since received FDA approval for the treatment of HR+/HER2- LABC/mBC may impact current 2 L rwPFS outcomes [ 61 , 62 ]. Finally, relative to the number of peer-reviewed full-text articles, this SLR identified eight abstracts and one poster presentation, comprising half (50%) of the included unique studies. As conference abstracts are inherently limited by how much content that can be described due to word limit constraints, this likely had implications on the present synthesis whereby we identified a dearth of real-world effectiveness outcomes in patients with HR+/HER2- LABC/mBC treated with 1 L CDK4/6i therapy.

Future research in this area should aim to address the limitations of the current literature and provide a more comprehensive understanding of optimal sequencing of effective and safe treatment for patients following 1 L CDK4/6i therapy. Specifically, future studies should strive to report robust data related to effectiveness, safety, and PROs for patients receiving 2 L treatment after 1 L CDK4/6i therapy. Future studies should also aim to understand the mechanism underlying CDK4/6i resistance. Addressing these gaps in knowledge may improve the long-term real-world management of patients with HR+/HER2- LABC/mBC. A future update of this synthesis may serve to capture a wider breadth of full-text, peer-reviewed articles to gain a more robust understanding of the safety, effectiveness, and real-world treatment patterns for patients with HR+/HER2- LABC/mBC. This SLR underscores the necessity for ongoing investigation and the development of innovative therapeutic approaches to address these gaps and improve patient outcomes.

This SLR qualitatively summarized the existing real-world effectiveness data for patients with HR+/HER2- LABC/mBC after 1 L CDK4/6i treatment. Results of this study highlight the limited available data and the suboptimal effectiveness of treatments employed in the 2 L setting and underscore the unmet need in this patient population. Additional studies reporting effectiveness and safety outcomes, in addition to PROs, for this patient population are necessary and should be the focus of future research.

PRISMA flow diagram. *Two included conference abstracts reported the same information as already included full-text reports, hence both conference abstracts were not identified as unique. Abbreviations: 1 L = first-line; AACR = American Association of Cancer Research; ASCO = American Society of Clinical Oncology; CDK4/6i = cyclin-dependent kinase 4/6 inhibitor; ESMO = European Society for Medical Oncology; ISPOR = Professional Society for Health Economics and Outcomes Research; n = number of studies; NMA = network meta-analysis; pts = participants; SABCS = San Antonio Breast Cancer Symposium; SLR = systematic literature review.

Number of studies reporting effectiveness outcomes exclusively for each treatment class. *Studies that lack sufficient information on effectiveness outcomes to classify based on the treatment classes outlined in the legend above. Abbreviations: CDK4/6i = cyclin-dependent kinase 4/6 inhibitor; ET = endocrine therapy; mTORi = mammalian target of rapamycin inhibitor.

Weighted average median rwPFS for 2 L treatments (recommended in ESMO/NCCN guidelines) after 1 L CDK4/6i treatment. Circular dot represents weighted average median across studies. Horizontal bars represent the range of values reported in these studies. Abbreviations: CDK4/6i = cyclin-dependent kinase 4/6 inhibitor; ESMO = European Society for Medical Oncology; ET = endocrine therapy, mTORi = mammalian target of rapamycin inhibitor; n = number of patients; NCCN = National Comprehensive Cancer Network; rwPFS = real-world progression-free survival.

Data availability

All data generated or analyzed during this study are included in this published article [and its supplementary information files]. This study is registered with PROSPERO (CRD42023383914).

Abbreviations

Second-line

Second-line treatment setting and beyond

American Association of Cancer Research

Aromatase inhibitor

American Society of Clinical Oncology

• Breast cancer

breast cancer gene/partner and localizer of BRCA2 positive

Clinical benefit rate

Cyclin-dependent kinase 4/6 inhibitor

Complete response

Duration of response

European Society for Medical Oncology

Food and Drug Administration

Human epidermal growth factor receptor 2

Human epidermal growth factor receptor 2 negative

Hormone receptor

Hormone receptor positive

Professional Society for Health Economics and Outcomes Research

Locally advanced breast cancer

Metastatic breast cancer

Medical Literature Analysis and Retrieval System Online

Medical subject headings

Mammalian target of rapamycin inhibitor

National Comprehensive Cancer Network

Newcastle Ottawa Scale

Objective response rate

Poly-ADP ribose polymerase inhibitor

Progression-free survival

Population, Intervention, Comparator, Outcome, Study Design

Partial response

Preferred Reporting Items for Systematic Literature Reviews and Meta-Analyses

Patient-reported outcomes

• Real-world evidence

San Antonio Breast Cancer Symposium

• Systematic literature review

Time-to-discontinuation

Time-to-next-treatment

Time-to-progression

United States

Łukasiewicz S, Czeczelewski M, Forma A, Baj J, Sitarz R, Stanisławek A, Breast, Cancer—Epidemiology. Risk factors, classification, prognostic markers, and current treatment Strategies—An. Updated Rev Cancers. 2021;13(17):4287.

Google Scholar  

World Health Organization (WHO). Breast Cancer Facts Sheet [updated July 12 2023. https://www.who.int/news-room/fact-sheets/detail/breast-cancer .

Arnold M, Morgan E, Rumgay H, Mafra A, Singh D, Laversanne M, et al. Current and future burden of breast cancer: global statistics for 2020 and 2040. Breast. 2022;66:15–23.

Article   PubMed   PubMed Central   Google Scholar  

Wilkinson L, Gathani T. Understanding breast cancer as a global health concern. Br J Radiol. 2022;95(1130):20211033.

Article   PubMed   Google Scholar  

Giaquinto AN, Sung H, Miller KD, Kramer JL, Newman LA, Minihan A et al. Breast Cancer Statistics, 2022. CA: A Cancer Journal for Clinicians. 2022;72(6):524– 41.

National Cancer Institute (NIH). Cancer Stat Facts: Female Breast Cancer [updated 2020. https://seer.cancer.gov/statfacts/html/breast.html .

American Cancer Society. Key Statistics for Breast Cancer [ https://www.cancer.org/cancer/types/breast-cancer/about/how-common-is-breast-cancer.html .

Zagami P, Carey LA. Triple negative breast cancer: pitfalls and progress. npj Breast Cancer. 2022;8(1):95.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Matutino A, Joy AA, Brezden-Masley C, Chia S, Verma S. Hormone receptor-positive, HER2-negative metastatic breast cancer: redrawing the lines. Curr Oncol. 2018;25(Suppl 1):S131–41.

Lloyd MR, Wander SA, Hamilton E, Razavi P, Bardia A. Next-generation selective estrogen receptor degraders and other novel endocrine therapies for management of metastatic hormone receptor-positive breast cancer: current and emerging role. Ther Adv Med Oncol. 2022;14:17588359221113694.

Cardoso F, Senkus E, Costa A, Papadopoulos E, Aapro M, André F, et al. 4th ESO-ESMO International Consensus guidelines for advanced breast Cancer (ABC 4)†. Ann Oncol. 2018;29(8):1634–57.

Article   CAS   PubMed   Google Scholar  

US Food Drug Administration. Palbociclib (Ibrance) 2017 [updated March 31, 2017. https://www.fda.gov/drugs/resources-information-approved-drugs/palbociclib-ibrance .

US Food Drug Administration. FDA expands ribociclib indication in HR-positive, HER2-negative advanced or metastatic breast cancer 2018 [updated July 18. 2018. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-expands-ribociclib-indication-hr-positive-her2-negative-advanced-or-metastatic-breast-cancer .

US Food Drug Administration. FDA approves abemaciclib for HR positive, HER2-negative breast cancer 2017 [updated Sept 28. 2017. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-abemaciclib-hr-positive-her2-negative-breast-cancer .

NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®). Breast Cancer 2022 [ https://www.nccn.org/professionals/physician_gls/pdf/breast.pdf .

Gennari A, André F, Barrios CH, Cortés J, de Azambuja E, DeMichele A, et al. ESMO Clinical Practice Guideline for the diagnosis, staging and treatment of patients with metastatic breast cancer. Ann Oncol. 2021;32(12):1475–95.

Beaver JA, Amiri-Kordestani L, Charlab R, Chen W, Palmby T, Tilley A, et al. FDA approval: Palbociclib for the Treatment of Postmenopausal Patients with estrogen Receptor-Positive, HER2-Negative metastatic breast Cancer. Clin Cancer Res. 2015;21(21):4760–6.

US Food Drug Administration. Ribociclib (Kisqali) [ https://www.fda.gov/drugs/resources-information-approved-drugs/ribociclib-kisqali#:~:text=On%20March%2013%2C%202017%2C%20the,hormone%20receptor%20(HR)%2Dpositive%2C .

US Food Drug Administration. FDA approves new treatment for certain advanced or metastatic breast cancers [ https://www.fda.gov/news-events/press-announcements/fda-approves-new-treatment-certain-advanced-or-metastatic-breast-cancers .

US Food Drug Administration. FDA expands ribociclib indication in HR-positive, HER2-negative advanced or metastatic breast cancer. 2018 [ https://www.fda.gov/drugs/resources-information-approved-drugs/fda-expands-ribociclib-indication-hr-positive-her2-negative-advanced-or-metastatic-breast-cancer .

US Food Drug Administration. FDA approves abemaciclib as initial therapy for HR-positive, HER2-negative metastatic breast cancer [ https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-abemaciclib-initial-therapy-hr-positive-her2-negative-metastatic-breast-cancer .

Turner NC, Slamon DJ, Ro J, Bondarenko I, Im S-A, Masuda N, et al. Overall survival with Palbociclib and fulvestrant in advanced breast Cancer. N Engl J Med. 2018;379(20):1926–36.

Slamon DJ, Neven P, Chia S, Fasching PA, De Laurentiis M, Im SA, et al. Phase III randomized study of Ribociclib and Fulvestrant in hormone Receptor-Positive, human epidermal growth factor receptor 2-Negative advanced breast Cancer: MONALEESA-3. J Clin Oncol. 2018;36(24):2465–72.

Goetz MP, Toi M, Campone M, Sohn J, Paluch-Shimon S, Huober J, et al. MONARCH 3: Abemaciclib as initial therapy for advanced breast Cancer. J Clin Oncol. 2017;35(32):3638–46.

Gopalan PK, Villegas AG, Cao C, Pinder-Schenck M, Chiappori A, Hou W, et al. CDK4/6 inhibition stabilizes disease in patients with p16-null non-small cell lung cancer and is synergistic with mTOR inhibition. Oncotarget. 2018;9(100):37352–66.

Watt AC, Goel S. Cellular mechanisms underlying response and resistance to CDK4/6 inhibitors in the treatment of hormone receptor-positive breast cancer. Breast Cancer Res. 2022;24(1):17.

Goetz M. MONARCH 3: final overall survival results of abemaciclib plus a nonsteroidal aromatase inhibitor as first-line therapy for HR+, HER2- advanced breast cancer. SABCS; 2023.

Munzone E, Pagan E, Bagnardi V, Montagna E, Cancello G, Dellapasqua S, et al. Systematic review and meta-analysis of post-progression outcomes in ER+/HER2– metastatic breast cancer after CDK4/6 inhibitors within randomized clinical trials. ESMO Open. 2021;6(6):100332.

Gennari A, André F, Barrios CH, Cortés J, de Azambuja E, DeMichele A, et al. ESMO Clinical Practice Guideline for the diagnosis, staging and treatment of patients with metastatic breast cancer. Annals of Oncology. 2021;32(12):1475-95.

European Society for Medical Oncology (ESMO). ESMO ​Metastatic Breast Cancer Living Guideline: ER-positive HER2-negative​ Breast Cancer​ [updated May 2023. https://www.esmo.org/living-guidelines/esmo-metastatic-breast-cancer-living-guideline/er-positive-her2-negative-breast-cancer .

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Welch PM VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). www.training.cochrane.org/handbook : Cochrane; 2021.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLoS Med. 2021;18(3):e1003583.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40–6.

Fraser C, Murray A, Burr J. Identifying observational studies of surgical interventions in MEDLINE and EMBASE. BMC Med Res Methodol. 2006;6(1):41.

US Food Drug Administration. Palbociclib (Ibrance). Silver Spring, MD: US Food and Drug Administration; 2017.

Book   Google Scholar  

GA Wells BS, D O’Connell J, Peterson V, Welch M, Losos PT. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses [ https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp .

Lo CK-L, Mertz D, Loeb M. Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments. BMC Med Res Methodol. 2014;14(1):45.

Martin JM, Handorf EA, Montero AJ, Goldstein LJ. Systemic therapies following progression on first-line CDK4/6-inhibitor treatment: analysis of real-world data. Oncologist. 2022;27(6):441–6.

Kalinsky KM, Kruse M, Smyth EN, Guimaraes CM, Gautam S, Nisbett AR et al. Abstract P1-18-37: Treatment patterns and outcomes associated with sequential and non-sequential use of CDK4 and 6i for HR+, HER2- MBC in the real world. Cancer Research. 2022;82(4_Supplement):P1-18-37-P1-18-37.

Choong GM, Liddell S, Ferre RAL, O’Sullivan CC, Ruddy KJ, Haddad TC, et al. Clinical management of metastatic hormone receptor-positive, HER2-negative breast cancer (MBC) after CDK 4/6 inhibitors: a retrospective single-institution study. Breast Cancer Res Treat. 2022;196(1):229–37.

Xi J, Oza A, Thomas S, Ademuyiwa F, Weilbaecher K, Suresh R, et al. Retrospective Analysis of Treatment Patterns and effectiveness of Palbociclib and subsequent regimens in metastatic breast Cancer. J Natl Compr Canc Netw. 2019;17(2):141–7.

Rozenblit M, Mun S, Soulos P, Adelson K, Pusztai L, Mougalian S. Patterns of treatment with everolimus exemestane in hormone receptor-positive HER2-negative metastatic breast cancer in the era of targeted therapy. Breast Cancer Res. 2021;23(1):14.

Bashour SI, Doostan I, Keyomarsi K, Valero V, Ueno NT, Brown PH, et al. Rapid breast Cancer Disease Progression following cyclin dependent kinase 4 and 6 inhibitor discontinuation. J Cancer. 2017;8(11):2004–9.

Giridhar KV, Choong GM, Leon-Ferre R, O’Sullivan CC, Ruddy K, Haddad T, et al. Abstract P6-18-09: clinical management of metastatic breast cancer (MBC) after CDK 4/6 inhibitors: a retrospective single-institution study. Cancer Res. 2019;79:P6–18.

Article   Google Scholar  

Mougalian SS, Feinberg BA, Wang E, Alexis K, Chatterjee D, Knoth RL, et al. Observational study of clinical outcomes of eribulin mesylate in metastatic breast cancer after cyclin-dependent kinase 4/6 inhibitor therapy. Future Oncol. 2019;15(34):3935–44.

Moscetti LML, Riggi L, Sperduti I, Piacentini FOC, Toss A, Barbieri E, Cortesi L, Canino FMA, Zoppoli G, Frassoldati A, Schirone A, Dominici MECF. SEQUENCE OF TREATMENTS AFTER CDK4/6 THERAPY IN ADVANCED BREAST CANCER (ABC), A GOIRC MULTICENTER RETRO/ PROSPECTIVE STUDY. PRELIMINARY RESULTS IN THE RETROSPECTIVE SERIES OF 116 PATIENTS. Tumori. 2022;108(4S):80.

Menichetti AZE, Giorgi CA, Bottosso M, Leporati R, Giarratano T, Barbieri C, Ligorio F, Mioranza E, Miglietta F, Lobefaro R, Faggioni G, Falci C, Vernaci G, Di Liso E, Girardi F, Griguolo G, Vernieri C, Guarneri V, Dieci MV. CDK 4/6 INHIBITORS FOR METASTATIC BREAST CANCER: A MULTICENTER REALWORLD STUDY. Tumori. 2022;108(4S):70.

Marschner NW, Harbeck N, Thill M, Stickeler E, Zaiss M, Nusch A, et al. 232P Second-line therapies of patients with early progression under CDK4/6-inhibitor in first-line– data from the registry platform OPAL. Annals of Oncology. 2022;33:S643-S4

Gousis C, Lowe KMH, Kapiris M. V. Angelis. Beyond First Line CDK4/6 Inhibitors (CDK4/6i) and Aromatase Inhibitors (AI) in Patients with Oestrogen Receptor Positive Metastatic Breast Cancer (ERD MBC): The Guy’s Cancer Centre Experience. Clinical Oncology2022. p. e178.

Endo Y, Yoshimura A, Sawaki M, Hattori M, Kotani H, Kataoka A, et al. Time to chemotherapy for patients with estrogen receptor-positive breast Cancer and cyclin-dependent kinase 4 and 6 inhibitor use. J Breast Cancer. 2022;25(4):296–306.

Li Y, Li W, Gong C, Zheng Y, Ouyang Q, Xie N, et al. A multicenter analysis of treatment patterns and clinical outcomes of subsequent therapies after progression on palbociclib in HR+/HER2- metastatic breast cancer. Ther Adv Med Oncol. 2021;13:17588359211022890.

Amaro CP, Batra A, Lupichuk S. First-line treatment with a cyclin-dependent kinase 4/6 inhibitor plus an aromatase inhibitor for metastatic breast Cancer in Alberta. Curr Oncol. 2021;28(3):2270–80.

Crocetti SPM, Tassone L, Marcantognini G, Bastianelli L, Della Mora A, Merloni F, Cantini L, Scortichini L, Agostinelli V, Ballatore Z, Savini A, Maccaroni E. Berardi R. What is the best therapeutic sequence for ER-Positive/HER2- Negative metastatic breast cancer in the era of CDK4/6 inhibitors? A single center experience. Tumori. 2020;106(2S).

Nichetti F, Marra A, Giorgi CA, Randon G, Scagnoli S, De Angelis C, et al. 337P Efficacy of everolimus plus exemestane in CDK 4/6 inhibitors-pretreated or naïve HR-positive/HER2-negative breast cancer patients: A secondary analysis of the EVERMET study. Annals of Oncology. 2020;31:S382

Luhn P, O’Hear C, Ton T, Sanglier T, Hsieh A, Oliveri D, et al. Abstract P4-13-08: time to treatment discontinuation of second-line fulvestrant monotherapy for HR+/HER2– metastatic breast cancer in the real-world setting. Cancer Res. 2019;79(4Supplement):P4–13.

Mittal A, Molto Valiente C, Tamimi F, Schlam I, Sammons S, Tolaney SM et al. Filling the gap after CDK4/6 inhibitors: Novel Endocrine and Biologic Treatment options for metastatic hormone receptor positive breast Cancer. Cancers (Basel). 2023;15(7).

Ashai N, Swain SM. Post-CDK 4/6 inhibitor therapy: current agents and novel targets. Cancers (Basel). 2023;15(6).

Higgins JPTTJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). www.training.cochrane.org/handbook : Cochrane; 2022.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Serdar CC, Cihan M, Yücel D, Serdar MA. Sample size, power and effect size revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochem Med (Zagreb). 2021;31(1):010502.

US Food Drug Administration. FDA approves elacestrant for ER-positive, HER2-negative, ESR1-mutated advanced or metastatic breast cancer [updated January 27 2023. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-elacestrant-er-positive-her2-negative-esr1-mutated-advanced-or-metastatic-breast-cancer .

US Food Drug Administration. FDA approves capivasertib with fulvestrant for breast cancer [updated November 16 2023. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-capivasertib-fulvestrant-breast-cancer .

Download references

Acknowledgements

The authors would like to acknowledge Joanna Bielecki who developed, conducted, and documented the database searches.

This study was funded by Pfizer Inc. (New York, NY, USA) and Arvinas (New Haven, CT, USA).

Author information

Sarah Kane, Belal Howidi, Bao-Ngoc Nguyen and Imtiaz A. Samjoo contributed equally to this work.

Authors and Affiliations

Pfizer, 10017, New York, NY, USA

Veronique Lambert & Yan Wu

EVERSANA, Burlington, ON, Canada

Sarah Kane, Belal Howidi, Bao-Ngoc Nguyen & Imtiaz A. Samjoo

Arvinas, 06511, New Haven, CT, USA

David Chandiwana & Michelle Edwards

You can also search for this author in PubMed   Google Scholar

Contributions

VL, IAS, SK, BH, BN, DC, YW, and ME participated in the conception and design of the study. IAS, SK, BH and BN contributed to the literature review, data collection, analysis, and interpretation of the data. VL, IAS, SK, BH, BN, DC, YW, and ME contributed to the interpretation of the data and critically reviewed for the importance of intellectual content for the work. VL, IAS, SK, BH, BN, DC, YW, and ME were responsible for drafting or reviewing the manuscript and for providing final approval. VL, IAS, SK, BH, BN, DC, YW, and ME meet the International Committee of Medical Journal Editors (ICMJE) criteria for authorship for this article, take responsibility for the integrity of the work, and have given their approval for this version to be published.

Corresponding author

Correspondence to Imtiaz A. Samjoo .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors of this manuscript declare that the research presented was funded by Pfizer Inc. and Arvinas. While the support from Pfizer Inc. and Arvinas was instrumental in facilitating this research, the authors affirm that their interpretation of the data and the content of this manuscript were conducted independently and without bias to maintain the transparency and integrity of the research. IAS, SK, BH, and BN are employees of EVERSANA, Canada, which was a paid consultant to Pfizer in connection with the development of this manuscript.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Lambert, V., Kane, S., Howidi, B. et al. Systematic literature review of real-world evidence for treatments in HR+/HER2- second-line LABC/mBC after first-line treatment with CDK4/6i. BMC Cancer 24 , 631 (2024). https://doi.org/10.1186/s12885-024-12269-8

Download citation

Received : 26 January 2024

Accepted : 16 April 2024

Published : 23 May 2024

DOI : https://doi.org/10.1186/s12885-024-12269-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• First-line CDK4/6i

ISSN: 1471-2407

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Help | Advanced Search

Computer Science > Computation and Language

Title: text generation: a systematic literature review of tasks, evaluation, and challenges.

Abstract: Text generation has become more accessible than ever, and the increasing interest in these systems, especially those using large language models, has spurred an increasing number of related publications. We provide a systematic literature review comprising 244 selected papers between 2017 and 2024. This review categorizes works in text generation into five main tasks: open-ended text generation, summarization, translation, paraphrasing, and question answering. For each task, we review their relevant characteristics, sub-tasks, and specific challenges (e.g., missing datasets for multi-document summarization, coherence in story generation, and complex reasoning for question answering). Additionally, we assess current approaches for evaluating text generation systems and ascertain problems with current metrics. Our investigation shows nine prominent challenges common to all tasks and sub-tasks in recent text generation publications: bias, reasoning, hallucinations, misuse, privacy, interpretability, transparency, datasets, and computing. We provide a detailed analysis of these challenges, their potential solutions, and which gaps still require further engagement from the community. This systematic literature review targets two main audiences: early career researchers in natural language processing looking for an overview of the field and promising research directions, as well as experienced researchers seeking a detailed view of tasks, evaluation methodologies, open challenges, and recent mitigation strategies.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .