systematic review of systematic review process research in software engineering

• DOI: 10.1016/j.infsof.2013.07.010
• Corpus ID: 260902727

A systematic review of systematic review process research in software engineering

• B. Kitchenham , P. Brereton
• Published in Information and Software… 1 December 2013
• Computer Science

Figures and Tables from this paper

708 Citations

Conducting systematic literature reviews and systematic mapping studies, reporting systematic reviews: some lessons from a tertiary study, quality assessment in systematic literature reviews: a software engineering perspective, guidelines for conducting systematic mapping studies in software engineering: an update.

• Highly Influenced

On the pragmatic design of literature studies in software engineering: an experience-based guideline

Defining protocols of systematic literature reviews in software engineering: a survey, tools to support systematic literature reviews in software engineering: a mapping study, enhancing the process specification for systematic literature reviews, segress: software engineering guidelines for reporting secondary studies, secondary studies in the academic context: a systematic mapping and survey, 53 references, systematic literature reviews in software engineering - a systematic literature review, systematic literature reviews in software engineering - a tertiary study, experiences using systematic review guidelines.

• Highly Influential

How Reliable Are Systematic Reviews in Empirical Software Engineering?

Lessons from applying the systematic literature review process within the software engineering domain, six years of systematic literature reviews in software engineering: an updated tertiary study.

• 33 Excerpts

Performing systematic literature reviews in software engineering

Repeatability of systematic literature reviews, systematic mapping studies in software engineering, related papers.

Showing 1 through 3 of 0 Related Papers

Research Repository

All Output Person Project

A systematic review of systematic review process in software engineering

Context Many researchers adopting systematic reviews (SRs) have also published papers discussing problems with the SR methodology and suggestions for improving it. Since guidelines for SRs in software engineering (SE) were last updated in 2007, we believe it is time to investigate whether the guidelines need to be amended in the light of recent research. Objective To identify, evaluate and synthesize research published by software engineering researchers concerning their experiences of performing SRs and their proposals for improving the SR process. Method We undertook a systematic review of papers reporting experiences of undertaking SRs and/or discussing techniques that could be used to improve the SR process. Studies were classified with respect to the stage in the SR process they addressed, whether they related to education or problems faced by novices and whether they proposed the use of textual analysis tools. Results We identified 68 papers reporting 63 unique studies published in SE conferences and journals between 2005 and mid-2012. The most common criticisms of SRs were that they take a long time, that SE digital libraries are not appropriate for broad literature searches and that assessing the quality of empirical studies of different types is difficult. Conclusion We recommend removing advice to use structured questions to construct search strings and including advice to use a quasi-gold standard based on a limited manual search to assist the construction of search stings and evaluation of the search process. Textual analysis tools are likely to be useful for inclusion/exclusion decisions and search string construction but require more stringent evaluation. SE researchers would benefit from tools to manage the SR process but existing tools need independent validation. Quality assessment of studies using a variety of empirical methods remains a major problem.

(2013). A systematic review of systematic review process in software engineering. Information and Software Technology, 2049 -2075. https://doi.org/10.1016/j.infsof.2013.07.010

B Kitchenham - A systematic review of systematic review process research in software engineering.pdf (867 Kb) PDF

Publisher Licence URL https://creativecommons.org/licenses/by-nc-nd/4.0/

Downloadable Citations

About Keele Repository

Administrator e-mail: [email protected]

This application uses the following open-source libraries:

SheetJS Community Edition

Apache License Version 2.0 ( http://www.apache.org/licenses/ )

Font Awesome

SIL OFL 1.1 ( http://scripts.sil.org/OFL )

MIT License ( http://opensource.org/licenses/mit-license.html )

CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/ )

Powered by Worktribe © 2024

Advanced Search

all of any of

Start your free trial

Arrange a trial for your organisation and discover why FSTA is the leading database for reliable research on the sciences of food and health.

REQUEST A FREE TRIAL

• Research Skills Blog

5 software tools to support your systematic review processes

By Dr. Mina Kalantar on 19-Jan-2021 13:01:01

Systematic reviews are a reassessment of scholarly literature to facilitate decision making. This methodical approach of re-evaluating evidence was initially applied in healthcare, to set policies, create guidelines and answer medical questions.

Systematic reviews are large, complex projects and, depending on the purpose, they can be quite expensive to conduct. A team of researchers, data analysts and experts from various fields may collaborate to review and examine incredibly large numbers of research articles for evidence synthesis. Depending on the spectrum, systematic reviews often take at least 6 months, and sometimes upwards of 18 months to complete.

The main principles of transparency and reproducibility require a pragmatic approach in the organisation of the required research activities and detailed documentation of the outcomes. As a result, many software tools have been developed to help researchers with some of the tedious tasks required as part of the systematic review process.

hbspt.cta._relativeUrls=true;hbspt.cta.load(97439, 'ccc20645-09e2-4098-838f-091ed1bf1f4e', {"useNewLoader":"true","region":"na1"});

The first generation of these software tools were produced to accommodate and manage collaborations, but gradually developed to help with screening literature and reporting outcomes. Some of these software packages were initially designed for medical and healthcare studies and have specific protocols and customised steps integrated for various types of systematic reviews. However, some are designed for general processing, and by extending the application of the systematic review approach to other fields, they are being increasingly adopted and used in software engineering, health-related nutrition, agriculture, environmental science, social sciences and education.

Software tools

There are various free and subscription-based tools to help with conducting a systematic review. Many of these tools are designed to assist with the key stages of the process, including title and abstract screening, data synthesis, and critical appraisal. Some are designed to facilitate the entire process of review, including protocol development, reporting of the outcomes and help with fast project completion.

As time goes on, more functions are being integrated into such software tools. Technological advancement has allowed for more sophisticated and user-friendly features, including visual graphics for pattern recognition and linking multiple concepts. The idea is to digitalise the cumbersome parts of the process to increase efficiency, thus allowing researchers to focus their time and efforts on assessing the rigorousness and robustness of the research articles.

This article introduces commonly used systematic review tools that are relevant to food research and related disciplines, which can be used in a similar context to the process in healthcare disciplines.

These reviews are based on IFIS' internal research, thus are unbiased and not affiliated with the companies.

This online platform is a core component of the Cochrane toolkit, supporting parts of the systematic review process, including title/abstract and full-text screening, documentation, and reporting.

The Covidence platform enables collaboration of the entire systematic reviews team and is suitable for researchers and students at all levels of experience.

From a user perspective, the interface is intuitive, and the citation screening is directed step-by-step through a well-defined workflow. Imports and exports are straightforward, with easy export options to Excel and CVS.

Access is free for Cochrane authors (a single reviewer), and Cochrane provides a free trial to other researchers in healthcare. Universities can also subscribe on an institutional basis.

Rayyan is a free and open access web-based platform funded by the Qatar Foundation, a non-profit organisation supporting education and community development initiative . Rayyan is used to screen and code literature through a systematic review process.

Unlike Covidence, Rayyan does not follow a standard SR workflow and simply helps with citation screening. It is accessible through a mobile application with compatibility for offline screening. The web-based platform is known for its accessible user interface, with easy and clear export options.

Function comparison of 5 software tools to support the systematic review process

EPPI-Reviewer

EPPI-Reviewer is a web-based software programme developed by the Evidence for Policy and Practice Information and Co-ordinating Centre  (EPPI) at the UCL Institute for Education, London .

It provides comprehensive functionalities for coding and screening. Users can create different levels of coding in a code set tool for clustering, screening, and administration of documents. EPPI-Reviewer allows direct search and import from PubMed. The import of search results from other databases is feasible in different formats. It stores, references, identifies and removes duplicates automatically. EPPI-Reviewer allows full-text screening, text mining, meta-analysis and the export of data into different types of reports.

There is no limit for concurrent use of the software and the number of articles being reviewed. Cochrane reviewers can access EPPI reviews using their Cochrane subscription details.

EPPI-Centre has other tools for facilitating the systematic review process, including coding guidelines and data management tools.

CADIMA is a free, online, open access review management tool, developed to facilitate research synthesis and structure documentation of the outcomes.

The Julius Institute and the Collaboration for Environmental Evidence established the software programme to support and guide users through the entire systematic review process, including protocol development, literature searching, study selection, critical appraisal, and documentation of the outcomes. The flexibility in choosing the steps also makes CADIMA suitable for conducting systematic mapping and rapid reviews.

CADIMA was initially developed for research questions in agriculture and environment but it is not limited to these, and as such, can be used for managing review processes in other disciplines. It enables users to export files and work offline.

The software allows for statistical analysis of the collated data using the R statistical software. Unlike EPPI-Reviewer, CADIMA does not have a built-in search engine to allow for searching in literature databases like PubMed.

DistillerSR

DistillerSR is an online software maintained by the Canadian company, Evidence Partners which specialises in literature review automation. DistillerSR provides a collaborative platform for every stage of literature review management. The framework is flexible and can accommodate literature reviews of different sizes. It is configurable to different data curation procedures, workflows and reporting standards. The platform integrates necessary features for screening, quality assessment, data extraction and reporting. The software uses Artificial Learning (AL)-enabled technologies in priority screening. It is to cut the screening process short by reranking the most relevant references nearer to the top. It can also use AL, as a second reviewer, in quality control checks of screened studies by human reviewers. DistillerSR is used to manage systematic reviews in various medical disciplines, surveillance, pharmacovigilance and public health reviews including food and nutrition topics. The software does not support statistical analyses. It provides configurable forms in standard formats for data extraction.

DistillerSR allows direct search and import of references from PubMed. It provides an add on feature called LitConnect which can be set to automatically import newly published references from data providers to keep reviews up to date during their progress.

The systematic review Toolbox is a web-based catalogue of various tools, including software packages which can assist with single or multiple tasks within the evidence synthesis process. Researchers can run a quick search or tailor a more sophisticated search by choosing their approach, budget, discipline, and preferred support features, to find the right tools for their research.

If you enjoyed this blog post, you may also be interested in our recently published blog post addressing the difference between a systematic review and a systematic literature review.

• FSTA - Food Science & Technology Abstracts
• IFIS Collections
• Resources Hub
• Diversity Statement
• Sustainability Commitment
• Company news
• Frequently Asked Questions
• Privacy Policy
• Terms of Use for IFIS Collections

Ground Floor, 115 Wharfedale Road,  Winnersh Triangle, Wokingham, Berkshire RG41 5RB

Get in touch with IFIS

© International Food Information Service (IFIS Publishing) operating as IFIS – All Rights Reserved     |     Charity Reg. No. 1068176     |     Limited Company No. 3507902     |     Designed by Blend

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Lessons from applying the systematic literature review process within the software engineering domain

2007, Journal of Systems and Software

Related Papers

Anayiaz Zaigmie

BTH, Sweden Official Website

Affan Yasin

Context: The Internet has become a vital channel for disseminating and accessing scientific literature for both the academic and industrial research needs. Nowadays, everyone has wide access to scientific literature repositories, which comprise of both “white” and “Grey” literature. The “Grey” literature, as opposed to “white” literature, is non-peer reviewed scientific information that is not available using commercial information sources such as IEEE or ACM. A large number of software engineering researchers are undertaking systematic literature reviews (SLRs) to investigate empirical evidence in software engineering. The key reason to include grey literature during information synthesis is to minimize the risk of any bias in the publication. Using the state of the art non-commercial databases that index information, the researchers can make the rigorous process of searching empirical studies in SLRs easier. This study explains the evidence of Grey literature while performing synthesis in Systematic Literature Reviews. Objectives: The goals of this thesis work are, 1. To identify the extent of usage of Grey Literature in synthesis during systematic literature reviews. 2. To investigate if non-commercial information sources primarily Google Scholar are sufficient for retrieving primary studies for SLRs. Methods: The work consists of a systematic literature review of SLRs and is a tertiary study and meta-analysis. The systematic literature review was conducted on 138 SLRs’ published through 2003 until 2012 (June). The article sources used are IEEEXplore, ACM Digital Library, Springer-Link and Science Direct. Results: For each of the selected article sources such as ACM, IEEEXplore, Springer-link and Science Direct, we have presented results, which describe the extent of the usage of Grey literature. The qualitative results discuss various strategies for systematic evaluation of the Grey literature during systematic literature review. The quantitative results comprise of charts and tables, showing the extent of Grey literature usage. The results from analysis of Google Scholar database describe the total number of primary studies that we are able to find using only Google Scholar database. Conclusion: From the analysis of 138 Systematic Literature Reviews (SLRs’), we conclude that the evidence of Grey literature in SLRs is around 9%. The percentage of Grey literature sources used in information synthesis sections of SLRs is around 93.2%. We were able to retrieve around 96 % of primary studies using Google Scholar database. We conclude that Google Scholar can be a choice for retrieving published studies however; it lacks detailed search options to target wider pool of articles. We also conclude that Grey literature is widely available in this age of information. We have provided guidelines in the form of strategies for systematic evaluation of Grey literature

EASE '14 Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Sandun Dasanayake

Context: Successfully addressing stakeholder concerns that are related to software system development and operation is crucial to achieving development goals. The importance of using a systematic approach to addressing these concerns throughout the software development life cycle is growing as more and more systems are employed to handle critical tasks. Objective: The goal of this study is to provide an overview of addressing concerns across the software development life cycle. Method: A systematic mapping study was conducted using a pre-defined protocol. Four digital databases were searched for research literature and primary studies were selected after a three round selection process conducted by multiple researchers. Results: The extracted data are processed and the results are reported from different viewpoints. The results are also analyzed against our research goals. Conclusion: We show that there is a considerable variation in the use of terminologies and addressing concerns in different phases of the software development life cycle.

Flaviana Amorim

Mahmood Niazi

Information and Software Technology

Matthias Galster , Sara Mahdavi-Hezavehi

Jerry Castro

First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007)

Torgeir Dingsøyr , Tore Dybå , Geir Hanssen

Olavo Barbosa

Software ecosystem is an approach that investigates the complex relationships among companies in the software industry. Companies work cooperatively and competitively in order to achieve their strategic objectives. They must engage in a new perspective considering both their own business and third party ones. Inspired from properties by natural and business ecosystems, a software ecosystem covers technical and business aspects of software development as well as partnership among companies. In this paper, we undertake a systematic mapping study to present a wide review of primary studies on software ecosystems. Systematic mapping is a methodology that gives, after a systematic research process, a visual summary map of its results. The search procedure identified 1026 studies, of which 44 were identified as relevant to answer our research questions. This study mapped what is currently known about software ecosystems perspective. We conclude that software ecosystems research is concentrated in 8 main areas in which the most relevant are open source software, ecosystem modeling, and business issues. The paper is intended to practitioners and academics investigating the field of software ecosystems. It contributes to summarize the body of knowledge in the field and direct efforts for future research in software ecosystems.

Journal of Software

Emília Mendes

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

Journal of Systems & Software

Denis Dennehy

Andre Santos

2016 SAI Computing Conference (SAI)

Pablo Alejandro Quezada-Sarmiento

ZILLE SUBHAN

Saif Programer

International Journal of Supply Chain Management

Hamzah Alaidaros

Luiz Fernando Capretz

Eduardo de Almeida

Majdi Abdellatief

Slamet Sucipto , Romi Satria Wahono Udinus

Fernando Paulovich

Rana Khallaf

Ricardo Alves

2011 Fifth International Workshop on Software Product Management (IWSPM)

Andrey Maglyas , Uolevi Nikula

European, Mediterranean & Middle Eastern Conference on Information Systems 2014 (EMCIS2014) October 27th – 28th 2014, Doha, Qatar

Negin Banaeianjahromi

Philipp Diebold

International Journal of Computer Science & Information Technology (IJCSIT) , George Valença

IEEE Transactions on Software Engineering

Richard Torkar , Lech Madeyski

Louis Major

Dietmar Winkler

Matthias Galster

IET Software

Hassan Gomaa

Dag I K Sjoberg

Chitu Okoli

Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering - EASE '14

Federica Sarro

Michel Chaudron

Per Runeson , Dag Sjøberg

Martin Solari , Dr. Gerardo Matturro

Barbara Russo

June Verner

Phoebe Chang

Francisco Vasconcellos , Geraldo Landre , Juliano Lopes de Oliveira

IAEME Publication

Emanoel Barreiros

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Systematic review on requirements engineering in quantum computing: insights and future directions.

1. Introduction

2. background, 2.1. quantum computing—general concepts, 2.2. milestones and current state in quantum computing, 2.3. software engineering and quantum software engineering, 2.4. importance of requirements engineering in software development, 2.5. requirements engineering and software quality, 2.6. requirements engineering and quantum requirements engineering, 2.7. the unique aspects of quantum computing that impact requirements engineering, 2.8. quantum computing and requirements engineering—case studies, 3. related work, 4. methodology, 4.1. slr protocol, 4.1.1. aim and need, 4.1.2. research questions, 4.1.3. search string.

• We extracted the keywords from the RQs.
• We considered synonyms for the keywords.
• We used the PICOC criterion to build the search string [ 72 ].

4.1.4. Inclusion and Exclusion Criteria

4.2. study selection and data extraction process, 4.2.1. search process.

• different versions of the same paper (we keep the last one),
• eliminate duplicates.
• Snowballing process, using the list of selected papers as seed [ 75 ].

4.2.2. Pilot Selection

4.2.3. data extraction protocol, 4.3. slr tool support, 5.1. rq1: what specific challenges are currently faced in re for qc, 5.1.1. specific re challenges on qc.

• Defining Quantum-Specific Requirements: Quantum-specific requirements are challenging due to the unique properties of quantum mechanics, such as superposition and entanglement, which complicate the definition of clear and actionable requirements.
• Hybrid System Requirements: Integrating quantum and classical components is particularly challenging. Careful balancing is required to ensure seamless interaction between both components. Requirements must specify data transfer and handle computational paradigm differences.
• Absence of Established Standards: A significant gap in standardized requirement engineering processes tailored for QC complicates the gathering, analysis, and validation of requirements.
• Continuous Update of Requirements: The rapid evolution of quantum technology necessitates continuous updates to requirement specifications to keep them relevant and accurate.
• Knowledge and Awareness: There needs to be an educational gap among requirements engineers regarding QC principles, limiting their ability to translate quantum capabilities into software requirements effectively.
• Requirement Specification Challenges: Defining and managing requirements for quantum systems is complex due to the unique properties of QC, such as superposition and entanglement, requiring new approaches to RE.
• Quantum-Specific Security Requirements: Addressing new security threats QC poses, such as vulnerabilities in classical cryptographic methods, requires specifying quantum-resistant cryptographic protocols.
• Testing and Verification: The fundamental differences between quantum and classical computing make developing effective testing and verification processes for quantum software requirements difficult. The difficulty in developing effective testing and verification processes can be attributed to the unique characteristics of QC requirements.

5.1.2. RE Challenges vs. ISO25010

5.1.3. other challenges.

• Hardware Challenges: The availability and stability of current quantum devices, such as NISQ systems, are inherently unstable, with high error and decoherence rates, which significantly limit the performance of quantum computations. Scaling quantum hardware to support fault-tolerant execution of quantum algorithms remains a critical challenge, including implementing robust error correction methods to handle the high error rates.
• Algorithm and Programming Challenges: Developing and optimizing efficient quantum algorithms is complex due to the need to handle quantum properties like superposition and entanglement. The lack of standardized and accessible quantum programming languages and the complexity of existing ones like Qiskit and Cirq pose significant obstacles for developers. Additionally, quantum programming requires developers to acquire new skills and knowledge, which can be lengthy and challenging due to the intrinsic complexity of quantum mechanics.
• Security Challenges: QC threatens classical cryptographic algorithms like RSA and ECC, which could become vulnerable to quantum attacks using algorithms like Shor’s. It is necessary to advance the creation of quantum-resistant cryptographic methods that can withstand potential quantum attacks. Integrating quantum-safe protocols into existing infrastructures is complex and requires considerations of compatibility and efficiency.
• Integration Challenges: Integrating quantum systems with classical computing infrastructures is a significant challenge due to fundamental differences in computational paradigms and the need to manage hybrid interfaces efficiently. Ensuring compatibility and effective integration between different quantum software platforms and quantum hardware is complex, especially given the rapid evolution of quantum technologies. The fast-paced evolution of quantum technology means that requirement specifications need continuous updates, presenting a significant challenge to maintain consistency and relevance.
• Quality and Maintenance Challenges: Ensuring the confidentiality and integrity of data in QC systems presents significant challenges. Testing and verifying quantum software is complex due to the fundamental differences between quantum and classical computing, necessitating new approaches and tools.

5.1.4. Answering the RQ1: What Specific Challenges Are Currently Faced in RE for QC?

• Complexity in Requirement Specification: Defining quantum-specific requirements is challenging due to the unique properties of quantum mechanics, such as superposition and entanglement. These properties make it difficult to define clear and actionable requirements for quantum software. Understanding how quantum operations differ fundamentally from classical ones is essential. For example, specifying how quantum algorithms should handle entangled states and superpositions in a way that aligns with expected outcomes is particularly complex.
• Integration with Classical Systems: Developing requirements for systems integrating both quantum and classical components presents significant challenges. These hybrid systems require a balance to ensure seamless interaction between both components. Requirements must specify how data will be transferred between quantum and classical systems and how to handle the differences in computational paradigms.
• Lack of Standardized Methodologies: A significant gap exists in standardized RE processes tailored for QC. This lack of established standards complicates the gathering, analyzing, and validating requirements. Developing standardized methods for eliciting requirements that consider quantum computational limitations and error rates is crucial.
• Rapid Technological Evolution: The fast-paced evolution of quantum technology necessitates continuous updates to requirement specifications to remain relevant and accurate. It is essential to keep requirements documents up-to-date with the latest quantum hardware capabilities and software improvements.
• Educational Gap: There is a significant educational gap among requirements engineers regarding QC principles. This gap limits their ability to translate quantum capabilities into software requirements effectively. Requirements engineers need training in quantum mechanics and QC concepts to develop accurate and feasible requirements.
• Security and Privacy Considerations: Quantum-specific security requirements are essential to address new security threats posed by QC, such as vulnerabilities in classical cryptographic methods. This challenge involves specifying requirements for integrating quantum-resistant cryptographic protocols to protect sensitive data within QC applications. The unique properties of QC necessitate that RE explicitly addresses these security concerns to ensure robust protection against quantum threats.
• Complexity of Testing and Validation: The fundamental differences between quantum and classical computing make developing effective testing and verification processes for quantum software requirements difficult. Defining test cases that can verify the correct implementation of quantum algorithms under the constraints of current quantum hardware is particularly challenging.

5.2. RQ2: What Opportunities Does QC Present for Advancements in RE?

5.2.1. specific re advances on qc.

• Specific Requirements Techniques for QC: Emerging techniques aim to adapt classical requirement engineering methods to the quantum context, defining quantum-specific functionalities more effectively. Advanced modeling tools are being developed to represent quantum algorithms and their integration with classical systems, aiding requirement analysts.
• Frameworks and Tools for Quantum Software RE: Developments in frameworks and tools facilitate the RE process for quantum software, addressing its unique needs. Integrating quantum and classical computing elements within software applications creates hybrid systems leveraging both paradigms’ strengths. Hybrid approaches combining classical and quantum methodologies bridge gaps between these realms. Standardization efforts and educational programs prepare requirement engineers for quantum-specific needs. New tools and frameworks enhance the precision and feasibility of modeling and specifying quantum software requirements.
• Hybrid RE Methods: Hybrid RE approaches combine classical and quantum methodologies, bridging gaps between these realms. Interdisciplinary collaborations lead to more robust strategies by incorporating insights from quantum physicists, software engineers, and blockchain experts, enhancing the overall requirement engineering process for quantum software.
• Requirements Modeling Techniques: Advances in modeling tools simulate quantum behaviors and integrate them with classical systems, providing new ways to specify quantum software requirements. Standardization efforts aim to create a common framework for these requirements. Educational programs prepare engineers with the necessary skills for handling quantum-specific needs. New tools and frameworks are being developed to enhance the precision and feasibility of specifying quantum software requirements.

5.2.2. RE Advances vs. ISO25010

5.2.3. other advances.

• Development of new quantum algorithms: Algorithms utilizing quantum superposition, entanglement, and interference have been introduced to enhance ensemble methods, allowing for exponential expansion of ensemble size with only a linear increase in quantum circuit depth. However, the real game-changer has been the quantum search algorithms. These algorithms, explicitly optimized for structured datasets, have improved search efficiency and reduced quantum resource consumption.
• Development of specific Quantum Programming Languages: Quantum programming languages such as Q#, Qiskit, and Silq have emerged as powerful tools, simplifying the implementation of quantum algorithms. These languages are specifically designed to handle the intricacies of quantum logic and computation, empowering engineers and developers to delve into the world of QC with confidence. Moreover, quantum software development frameworks and SDKs provide a wealth of resources, enabling engineers and developers to experiment with and implement quantum algorithms more efficiently and effectively.
• Quantum simulation tools and techniques: These advances include the development of parametric compilation and active qubit reset methods, which allow rapid adjustments of quantum program parameters without complete recompilation and reduce latency by quickly resetting qubits to their ground state between computations. Additionally, modeling and simulation tools that integrate quantum behaviors with classical systems have progressed, offering new ways to visualize and specify requirements for quantum software.
• Quantum Hardware Development: Advances in quantum hardware design, such as increased qubit coherence times and improved qubit connectivity, allow for overcoming current limitations and enabling more efficient quantum computations. Additionally, innovations in the architectural design of QRAM enable more scalable and efficient designs, overcoming some of the current physical limitations in QC hardware.
• Quantum Information Theory: Quantum-resistant cryptographic algorithms have been developed to withstand attacks from both quantum and classical computers. These include lattice-based, hash-based, multivariate, and code-based cryptographic schemes. Many initiatives seek to standardize quantum-resistant cryptographic protocols, ensuring a robust and unified approach to protecting data against quantum threats.
• Practical Applications of QC in Machine Learning and Security: Developing specialized architectures for quantum machine learning supports the entire lifecycle of development, from data handling to model deployment. Implementations such as quantum reinforcement learning and quantum neural networks demonstrate the practical applicability of these architectures. Furthermore, quantum key distribution (QKD) and quantum-resistant algorithms safeguard IoT systems against emerging quantum threats.
• Interdisciplinary Collaborations and Research Projects: Collaboration among computer scientists, physicists, and engineers is leading to more comprehensive approaches in quantum requirements engineering (QRE), integrating knowledge from various disciplines to improve requirements gathering and analysis processes. Additionally, the integration of quantum cryptography and blockchain technology offers a dual layer of security, leveraging the unconditional security of quantum key distribution and the immutability of blockchain.

5.2.4. Answering RQ2: What Opportunities Does QC Present for Advancements in RE?

• Enhanced Specification Techniques: QC requires new RE techniques to address unique properties such as qubits and entanglement. Developing quantum-specific requirements ensures these aspects are accurately captured and managed. Adapting classical methods to define quantum functionalities helps align software with quantum operational paradigms. The following is an overview of the techniques used for RE within quantum software development projects based on the authors’ claims in the selected papers. To elicit information, the authors propose practical techniques such as interviews and questionnaires, literature reviews, and the analysis of repositories such as Stack Exchange and GitHub. Taxonomies, UML, and Q-UML can be mentioned for modeling and representing the problem domain. User stories are specifically mentioned for the system’s functionalities. In the particular case of non-functional requirements, the use of ISO-9126 [ 78 ] and ISO-25010 [ 47 ] quality frameworks is mentioned. Finally, agile QC practices and techniques, the Quingo framework, and the Talavera Manifesto, among others, are mentioned as reference frameworks that guide not only the RE but also the SE of the entire project.
• Improved Modeling and Simulation Tools: The complexity of quantum algorithms necessitates advanced modeling tools to represent quantum behaviors and their integration with classical systems. These tools enhance understanding and specification of requirements, facilitating the development of robust quantum software. Integrating quantum behaviors into modeling tools improves precision in requirement specification.
• Hybrid Requirement Engineering Approaches: Combining classical and quantum methodologies bridges the gap between the two paradigms, enabling comprehensive requirement engineering processes. Hybrid approaches leverage the strengths of both computing types, ensuring effective and integrated solutions. These methods address the unique challenges of both quantum and classical systems.
• Interdisciplinary Collaboration: QC is a field that necessitates collaboration among computer scientists, physicists, and engineers. Diverse expertises and perspectives are crucial in these interdisciplinary efforts, leading to robust requirement engineering strategies. This collaboration enhances the quality and completeness of requirement specifications.
• Standardization Efforts: Standardizing quantum software development practices, including requirement engineering processes, is a crucial step. It creates common frameworks for specifying and validating requirements, ensuring consistency and reliability across quantum software projects. These standardized practices foster high quality and interoperability in quantum-classical applications, providing a solid foundation for our work.
• Educational Programs: Educational programs and resources in QC equip requirement engineers with the knowledge to handle quantum-specific requirements. This training bridges the educational gap and fosters a skilled workforce capable of advancing RE in quantum contexts. Enhanced education ensures accurate specification of quantum software requirements.
• Security and Cryptography: Developing quantum-resistant cryptographic algorithms ensures robust security requirements against QC threats. This advancement is crucial for specifying quantum-safe security measures. Quantum-resistant schemes provide a foundation for secure requirements, protecting sensitive data from quantum threats.

5.3. RQ3: What Future Directions or Trends Are Emerging in the Field of RE Specific to QC?

5.3.1. specific re future directions on qc, 5.3.2. re future directions vs. iso25010, 5.3.3. other future research directions.

• Quantum Algorithms: Future research in quantum algorithms focuses on several key areas. Firstly, there is significant interest in optimizing algorithms to reduce quantum resource requirements and enhance efficiency, particularly for NP-complete problems and molecular simulation. Additionally, quantum algorithms are pushed to be extended to more complex and diverse problems, such as network optimization and energy systems. Integrating quantum algorithms with classical systems to maximize efficiency and applicability is also a crucial research direction.
• Quantum Hardware: Future research in quantum hardware highlights the need to improve qubits’ stability and coherence times, essential for reducing errors during quantum operations. There is also a focus on developing robust quantum hardware that can operate under real-world conditions and exploring new materials to construct more stable qubit systems. Additionally, integrating quantum hardware with classical systems to create scalable quantum-classical hybrid systems is a crucial area of interest.
• Error Correction: Error correction is a critical aspect of QC, and future research aims to develop more efficient and less resource-intensive error correction techniques. These techniques include exploring advanced quantum error correction methods and integrating them into the quantum software development lifecycle to manage inherent quantum errors more effectively.
• Quantum Security: Future research in quantum security focuses on optimizing quantum-resistant algorithms to enhance their efficiency and scalability. There is also a significant emphasis on developing comprehensive security frameworks that integrate quantum capabilities into existing IoT systems and other platforms, ensuring robust protection against classical and quantum threats.
• Applications in Physics and Chemistry: Future research directions for applications in physics and chemistry include developing quantum algorithms tailored for specific domains like molecular simulation and material science. These efforts aim to broaden the scope of QC applications and improve the accuracy and efficiency of simulations in these fields.
• Optimization and Logistics: Research in optimization and logistics focuses on enhancing the capabilities of quantum algorithms to solve complex scheduling and optimization problems. This optimization includes the development of hybrid algorithms that dynamically allocate tasks between quantum and classical systems to leverage their respective strengths.
• User Interface and Software Tools: Future research in user interfaces and software tools involves creating more sophisticated virtual instrument interfaces and enhancing real-time data processing capabilities. Developing robust tools seamlessly integrating quantum and classical computing elements is also a significant research focus.
• Education and Training in QC: Addressing the growing need for skilled professionals in QC is crucial. Future research emphasizes the development of comprehensive educational programs and training materials. These programs aim to equip engineers and developers with the necessary skills to navigate the complexities of QSE and integrate QC technologies effectively.

5.3.4. Answering the RQ3: What Future Directions or Trends Are Emerging in the Field of RE Specific to QC?

• Formalizing QRE Processes: Future research emphasizes formalizing the processes specific to quantum requirement engineering. This formalization includes developing standardized methodologies for specifying, modeling, and analyzing quantum software requirements. These methodologies aim to address the unique characteristics of QC, such as superposition and entanglement, which are not present in classical computing. The critical research focus involves developing comprehensive frameworks for quantum requirement engineering, creating standardized practices to ensure consistency and quality across projects and employing formal methods to handle quantum-specific phenomena.
• Development of Validation and Verification Tools: Another emerging trend is the development of sophisticated tools and frameworks for validating and verifying quantum software requirements. These tools ensure that the requirements specified are correct, complete, and feasible given the current state of quantum hardware and algorithms. Critical research efforts are directed towards building tools to validate quantum software requirements, ensuring alignment with intended functionalities, and leveraging simulations to test requirements against quantum phenomena.
• Standardizing Architectural Practices: There is a push towards standardizing architectural practices and methodologies specific to quantum software. This helps create mature, reliable, and fault-tolerant quantum systems, ensuring their stability and performance. The primary research objectives include developing best practices and standardized architectural methodologies and promoting consistency and integration efficiency across different quantum software projects.
• Methodologies for Dynamic Requirement Updates and Interdisciplinary Collaboration: QC evolves rapidly, necessitating methodologies that support dynamic requirements updates and encourage interdisciplinary collaboration. This trend focuses on creating adaptive tools and processes to handle the fast-paced advancements in quantum technologies. Key focus areas include developing tools for dynamic requirement updates and encouraging collaboration between quantum physicists, software engineers, and requirement engineers.
• Creation of specific QRE Methodologies: It is crucial to develop methodologies tailored specifically to QC. These methodologies must address QC’s probabilistic nature and rapid technological evolution. The primary research focus involves creating specific methodologies for quantum requirements and continuously refining approaches to keep pace with technological advancements.
• Sophisticated Modeling Tools: The development of advanced modeling tools for quantum requirements is essential. These tools should provide accurate modeling capabilities that account for the complex behaviors of quantum systems. Key research areas include enhancing existing modeling tools, creating new domain-specific languages, and providing tools that can effectively model and test quantum requirements.
• Educational Programs for QRE: There is a significant need for educational programs that equip engineers with the necessary skills and knowledge to handle quantum requirements. This trend focuses on developing curricula, workshops, and certification programs to bridge the knowledge gap. Critical efforts involve establishing comprehensive educational and training programs and equipping engineers with QC principles and practices.
• Automated Tools for RE: Another emerging trend is the development of automated tools to support the RE process. These tools aim to reduce manual effort and increase accuracy. Research focuses on creating AI-driven tools for requirement engineering and incorporating simulations and testing frameworks.
• Hybrid Modeling Techniques: Innovating hybrid modeling techniques that integrate quantum and classical computing elements is essential for seamless requirement modeling. This trend focuses on bridging the gap between different computational paradigms. Key research areas include developing hybrid models integrating quantum and classical computing and refining integration techniques for practical co-existence.
• Defined Software Development Lifecycles: Establishing well-defined software development lifecycles specific to quantum applications is crucial. This trend focuses on creating structured frameworks for developing, testing, and maintaining quantum software. The key focus areas include defining lifecycle phases for quantum software and integrating requirement engineering, testing, and maintenance processes.
• Improved Methodological Support: Enhancing methodological support for RE in QC is necessary to handle the complexities of quantum systems. This improvement involves refining existing methodologies and creating new approaches tailored to QC. The research aims to develop robust methodologies for quantum RE and create new approaches to address quantum-specific challenges.

5.4. Trends and Cataloging

Publishers and sources, temporal analysis, 5.5. strenght of evidence, 6. discussion, 6.1. interpreting the answers to rqs, 6.1.1. interpreting the answer to rq1.

• Inherent complexity of quantum systems,
• Lack of specific tools and methodologies,
• Uncertainty in the behavior of quantum systems,
• Standardized approaches for quantum systems.

6.1.2. Interpreting the Answer to RQ2

6.1.3. interpreting the answer to rq3, 6.1.4. quality attributes on quantum computing, 6.1.5. security and qre, 6.1.6. summary, 6.2. bibliometric analysis, 6.2.1. keywords and relevant concepts, 6.2.2. relationship—most relevant terms, 6.2.3. relationship—most relevant authors, 6.3. threats to validity, 6.3.1. descriptive validity.

• The information to be collected was structured using various forms of data extraction through a Google Sheets data spreadsheet to support uniform data recording and ensure the objectivity of the data extraction process.
• We held weekly meetings to unify critical concepts with the research and classification criteria, answer any questions, and demonstrate how to carry out the process.

6.3.2. Theoretical Validity

• We built a search string and adapted it to the data sources defined.
• We defined exclusion and inclusion criteria to guarantee objectivity in the selection process.
• We performed cross-checks among researchers to visualize the criteria’s applicability.
• Including articles written in English and discarding studies in other languages could have a minimal impact on this criterion.
• We expanded the scope of the study with a first snowballing search review, according to Wohlin’s guidelines [ 75 ], obtaining eight additional papers for the study.

6.3.3. Generalizability

• We ensured that the scope of RQs was broad enough to identify and classify results on different QC and RE approaches, regardless of specific cases and domains, among others.

6.3.4. Interpretive Validity

• All researchers reviewed and validated the conclusions of the study.
• A researcher with expertise in the RE area assisted us in interpreting the data.

6.3.5. Repeatability

• We previously published an initial protocol through the arXiv platform [ 71 ].
• We designed a detailed protocol (see Section 4 ) so other researchers can repeat the process and corroborate the results.
• We validated the structure of our report using the 2020 PRISMA Statement (Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA), https://www.prisma-statement.org/ (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). This statement is a checklist that contains the minimum set of items that should be reported in SLR and meta-analyses [ 98 ]. We accomplished all the items except those assessing quantitative survey data and comparing them (items n° 13a-13f, 19, 20a-20d, and 24a).

7. Conclusions

Supplementary materials, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest, abbreviations.

Appendix A. Detailed RQs and Answers from Related Work

Appendix B. Selected Papers—Final List

• Haghparast, M.; Mikkonen, T.; Nurminen, J.K.; Stirbu, V. Quantum Software Engineering Challenges from Developers’ Perspective: Mapping Research Challenges to the Proposed Workflow Model. In Proceedings of the 2023 IEEE International Conference on Quantum Computing and Engineering (QCE), Bellevue, WA, USA, 17–22 September 2023; IEEE: Piscataway, NJ, USA, 2023; Volume 2, pp. 173–176. [ Google Scholar ]
• Sun, Z.; Gao, W.; Dong, H.; Xie, H.; Yang, L. A new post-quantum voting protocol based on physical laws. Quantum Inf. Process. 2022 , 21 , 289. [ Google Scholar ] [ CrossRef ]
• Sood, S.K.; Pooja. Quantum computing review: A decade of research. IEEE Trans. Eng. Manag. 2023 , 71 , 6662–6676. [ Google Scholar ] [ CrossRef ]
• Yue, T.; Mauerer, W.; Ali, S.; Taibi, D. Challenges and Opportunities in Quantum Software Architecture. In Software Architecture: Research Roadmaps from the Community ; Springer: Cham, Switzerland, 2023; pp. 1–23. [ Google Scholar ]
• Piattini, M.; Serrano, M.; Perez-Castillo, R.; Petersen, G.; Hevia, J.L. Toward a quantum software engineering. IT Prof. 2021 , 23 , 62–66. [ Google Scholar ] [ CrossRef ]
• Ali, O.; Abdelbaki, W.; Shrestha, A.; Elbasi, E.; Alryalat, M.A.A.; Dwivedi, Y.K. A systematic literature review of artificial intelligence in the healthcare sector: Benefits, challenges, methodologies, and functionalities. J. Innov. Knowl. 2023 , 8 , 100333. [ Google Scholar ] [ CrossRef ]
• Zhao, J. Quantum software engineering: Landscapes and horizons. arXiv 2020 , arXiv:2007.07047. [ Google Scholar ]
• Zanni Junior, P.E.; Vieira de Camargo, V. A systematic mapping on quantum software development in the context of software engineering. arXiv 2021 , arXiv:2106.00926. [ Google Scholar ]
• Cartiere, C.R. Formal Methods for Quantum Software Engineering. In Quantum Software Engineering ; Springer: Berlin/Heidelberg, Germany, 2022; pp. 85–101. [ Google Scholar ]
• Barbosa, L.S. Software engineering for ‘quantum advantage’. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, Seoul, Republic of Korea, 27 June–19 July 2020; pp. 427–429. [ Google Scholar ]
• Weder, B.; Barzen, J.; Leymann, F.; Vietz, D. Quantum software development lifecycle. In Quantum Software Engineering ; Springer: Berlin/Heidelberg, Germany, 2022; pp. 61–83. [ Google Scholar ]
• Pérez-Castillo, R.; Serrano, M.A.; Piattini, M. Software modernization to embrace quantum technology. Adv. Eng. Softw. 2021 , 151 , 102933. [ Google Scholar ] [ CrossRef ]
• Akbar, M.A.; Khan, A.A.; Mahmood, S.; Rafi, S. Quantum software engineering: A new genre of computing. arXiv 2022 , arXiv:2211.13990. [ Google Scholar ]
• Saraiva, L.; Haeusler, E.H.; Costa, V.G.; Kalinowski, M. Non-Functional Requirements for Quantum Programs. In Proceedings of the Q-SET@ QCE, Broomfield, CO, USA, 17–22 October 2021; pp. 89–73. [ Google Scholar ]
• Benioff, P. The computer as a physical system: A microscopic quantum mechanical Hamiltonian model of computers as represented by Turing machines. J. Stat. Phys. 1980 , 22 , 563–591. [ Google Scholar ] [ CrossRef ]
• Toffoli, T. Reversible computing. In Proceedings of the International Colloquium on Automata, Languages, and Programming, Noordweijkerhout, The Netherlands, 14–18 July 1980; Springer: Berlin/Heidelberg, Germany, 1980; pp. 632–644. [ Google Scholar ]
• Feynman, R.P. Simulating physics with computers. Int. J. Theor. Phys. 1982 , 21 , 467–488. [ Google Scholar ] [ CrossRef ]
• Deutsch, D.; Penrose, R. Quantum theory, the Church–Turing principle and the universal quantum computer. Proc. R. Soc. Lond. A Math. Phys. Sci. 1985 , 400 , 97–117. [ Google Scholar ] [ CrossRef ]
• Vedral, V. Introduction to Quantum Information Science ; Oxford University Press: Oxford, UK, 2006. [ Google Scholar ]
• Horodecki, R.; Horodecki, P.; Horodecki, M.; Horodecki, K. Quantum entanglement. Rev. Mod. Phys. 2009 , 81 , 865. [ Google Scholar ] [ CrossRef ]
• Deutsch, D.; Jozsa, R. Rapid Solution of Problems by Quantum Computation. Proc. R. Soc. Lond. Ser. A 1992 , 439 , 553–558. [ Google Scholar ] [ CrossRef ]
• Simon, D.R. On the Power of Quantum Computation. SIAM J. Comput. 1997 , 26 , 1474–1483. [ Google Scholar ] [ CrossRef ]
• Shor, P. Algorithms for quantum computation: Discrete logarithms and factoring. In Proceedings of the 35th Annual Symposium on Foundations of Computer Science, Santa Fe, NM, USA, 20–22 November 1994; pp. 124–134. [ Google Scholar ] [ CrossRef ]
• Grover, L.K. A fast quantum mechanical algorithm for database search. In Proceedings of the STOC ’96: Twenty-Eighth Annual ACM Symposium on Theory of Computing, Philadelphia, PA, USA, 22–24 May 1996; pp. 212–219. [ Google Scholar ] [ CrossRef ]
• Quantum Computing|D-Wave—dwavesys.com. Available online: https://www.dwavesys.com/learn/quantum-computing/ (accessed on 2 June 2024).
• IBM Quantum Computing—ibm.com. Available online: https://www.ibm.com/quantum (accessed on 2 June 2024).
• IBM Quantum System Two: The Era of Quantum Utility is Here|IBM Quantum Computing Blog—ibm.com. Available online: https://www.ibm.com/quantum/blog/quantum-roadmap-2033 (accessed on 2 June 2024).
• Russell, J. IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans—hpcwire.com. Available online: https://www.hpcwire.com/2019/01/10/ibm-quantum-update-q-system-one-launch-new-collaborators-and-qc-center-plans/ (accessed on 2 June 2024).
• Servicio de Computación Cuántica en la Nube—Amazon Braket—AWS—aws.amazon.com. Available online: https://aws.amazon.com/es/braket/ (accessed on 2 June 2024).
• Luo, J.; Zhao, P.; Miao, Z.; Lan, S.; Zhao, J. A Comprehensive Study of Bug Fixes in Quantum Programs. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Honolulu, HI, USA, 15-18 March 2022; pp. 1239–1246. [ Google Scholar ] [ CrossRef ]
• Yang, Z.; Zolanvari, M.; Jain, R. A survey of important issues in quantum computing and communications. IEEE Commun. Surv. Tutorials 2023 , 25 , 1059–1094. [ Google Scholar ] [ CrossRef ]
• IEEE Std 610.12-1990 ; IEEE Standard Glossary of Software Engineering Terminology. IEEE: Piscataway, NJ, USA, 1990.
• Bourque, P.; Fairley, R.E. Guide to the Software Engineering Body of Knowledge-SWEBOK: v3.0 ; IEEE Computer Society Press: Washington, DC, USA, 2014. [ Google Scholar ]
• Brooks, F.P., Jr. No Silver Bullet Essence and Accidents of Software Engineering. Computer 1987 , 20 , 10–19. [ Google Scholar ] [ CrossRef ]
• Vukšić, V.B.; Ivančić, L.; Vugec, D.S. A Preliminary Literature Review of Digital Transformation Case Studies. Int. J. Comput. Inf. Eng. 2018 , 12 , 737–742. [ Google Scholar ]
• Felderer, M.; Taibi, D.; Palomba, F.; Epping, M.; Lochau, M.; Weder, B. Software Engineering Challenges for Quantum Computing: Report from the First Working Seminar on Quantum Software Engineering (WSQSE 22). ACM SIGSOFT Softw. Eng. Notes 2023 , 48 , 29–32. [ Google Scholar ] [ CrossRef ]
• Carleton, A.; Klein, M.; Robert, J.; Harper, E. Architecting the Future of Software Engineering. A National Agenda for Software Engineering Research & Development ; Technical Report; Carnegie Mellon University, Software Engineering Institute: Pittsburgh, PA, USA, 2021. [ Google Scholar ]
• De Stefano, M.; Pecorelli, F.; Di Nucci, D.; Palomba, F.; De Lucia, A. Software engineering for quantum programming: How far are we? J. Syst. Softw. 2022 , 190 , 111326. [ Google Scholar ] [ CrossRef ]
• Alexeev, Y.; McCaskey, A.; De Jong, W. Introduction to the Special Issue on Software Tools for Quantum Computing: Part 1. ACM Trans. Quantum Comput. 2022 , 3 , 1–3. [ Google Scholar ] [ CrossRef ]
• Greiwe, F.; Krüger, T.; Mauerer, W. Effects of imperfections on quantum algorithms: A software engineering perspective. In Proceedings of the 2023 IEEE International Conference on Quantum Software (QSW), Chicago, IL, USA, 2–8 July 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 31–42. [ Google Scholar ]
• Serrano, M.A.; Pérez-Castillo, R.; Piattini, M. Quantum Software Engineering ; Springer Nature: Berlin/Heidelberg, Germany, 2022. [ Google Scholar ]
• Boehm, B. Software Engineering Economics ; Prentice Hall: Upper Saddle River, NJ, USA, 1981. [ Google Scholar ]
• Daneva, M.; Pastor, O. Requirements Engineering: Foundation for Software Quality ; Lecture Notes in Computer Science; Springer International Publishing: Cham, Switzerland, 2016; Volume 9619. [ Google Scholar ] [ CrossRef ]
• Noorin, U.; Sirshar, M. Quality Assurance in Requirement Engineering. Glob. J. Comput. Sci. Technol. 2017 , 17 , V1. [ Google Scholar ]
• Srivastava, P.R.; Kumar, K. An Approach towards Software Quality Assessment. In Information Systems, Technology and Management ; Lecture Notes in Computer Science; Springer International Publishing: Cham, Switzerland, 2009; pp. 150–160. [ Google Scholar ] [ CrossRef ]
• Hofmann, H.F.; Lehner, F. Requirements Engineering as a Success Factor in Software Projects. IEEE Softw. 2001 , 18 , 58–66. [ Google Scholar ] [ CrossRef ]
• Standard ISO/IEC 25010:2023 ; Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—Product Quality Model. International Organization for Standardization: Geneva, Switzerland, 2023.
• Marir, T.; Mokhati, F.; Seridi-Bouchelaghem, H. Do we need specific quality models for multi-agent systems? Toward using the ISO/IEC 25010 quality model for MAS. In Proceedings of the 2014 9th International Conference on Software Engineering and Applications (ICSOFT-EA), Vienna, Austria, 29–31 August 2014; pp. 363–368. [ Google Scholar ] [ CrossRef ]
• Falcó, M.; Robiolo, G. Building a Catalogue of ISO/IEC 25010 Quality Measures Applied in an Industrial Context. J. Phys. Conf. Ser. 2021 , 1828 , 012077. [ Google Scholar ] [ CrossRef ]
• Murillo, J.M.; Garcia-Alonso, J.; Moguel, E.; Barzen, J.; Leymann, F.; Ali, S.; Yue, T.; Arcaini, P.; Pérez, R.; de Guzmán, I.G.R.; et al. Challenges of Quantum Software Engineering for the Next Decade: The Road Ahead. arXiv 2024 , arXiv:2404.06825. [ Google Scholar ]
• Yue, T.; Ali, S.; Arcaini, P. Towards Quantum Software Requirements Engineering. In Proceedings of the 2023 IEEE International Conference on Quantum Computing and Engineering (QCE), Bellevue, WA, USA, 17–22 September 2023; IEEE: Piscataway, NJ, USA, 2023; Volume 2, pp. 161–164. [ Google Scholar ]
• Jantunen, S.; Dumdum, R.; Gause, D. Towards new requirements engineering competencies. In Proceedings of the 2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), Montreal, QC, Canada, 27 May 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 131–134. [ Google Scholar ]
• Park, J.L. The Concept of Transition in Quantum Mechanics. Found. Phys. 1970 , 1 , 23–33. [ Google Scholar ] [ CrossRef ]
• Kumar Pati, A.; Braunstein, S.L. Impossibility of deleting an unknown quantum state. Nature 2000 , 404 , 164–165. [ Google Scholar ] [ CrossRef ]
• Nielsen, M.A.; Chuang, I.L. Programmable Quantum Gate Arrays. Phys. Rev. Lett. 1997 , 79 , 321–324. [ Google Scholar ] [ CrossRef ]
• Bae, J.; Kwek, L.C. Quantum state discrimination and its applications. J. Phys. A Math. Theor. 2015 , 48 , 083001. [ Google Scholar ] [ CrossRef ]
• System information|IBM Quantum Documentation—docs.quantum.ibm.com. Available online: https://docs.quantum.ibm.com/run/system-information (accessed on 2 June 2024).
• Holmes, A.; Johri, S.; Guerreschi, G.G.; Clarke, J.S.; Matsuura, A.Y. Impact of qubit connectivity on quantum algorithm performance. Quantum Sci. Technol. 2020 , 5 , 025009. [ Google Scholar ] [ CrossRef ]
• Yuan, P.; Allcock, J.; Zhang, S. Does Qubit Connectivity Impact Quantum Circuit Complexity? IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 2024 , 43 , 520–533. [ Google Scholar ] [ CrossRef ]
• Cowtan, A.; Dilkes, S.; Duncan, R.; Krajenbrink, A.; Simmons, W.; Sivarajah, S. On the Qubit Routing Problem. arXiv 2019 . [ Google Scholar ] [ CrossRef ]
• Shor, P. Fault-tolerant quantum computation. In Proceedings of the 37th Conference on Foundations of Computer Science, Burlington, VT, USA, 14–16 October 1996; pp. 56–65. [ Google Scholar ] [ CrossRef ]
• Devitt, S.J.; Munro, W.J.; Nemoto, K. Quantum error correction for beginners. Rep. Prog. Phys. 2013 , 76 , 076001. [ Google Scholar ] [ CrossRef ]
• Krüger, T.; Mauerer, W. Quantum Annealing-Based Software Components: An Experimental Case Study with SAT Solving. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, Seoul, Republic of Korea, 27 June 2020–19 July 2020; pp. 445–450. [ Google Scholar ] [ CrossRef ]
• Spoletini, P. Towards Quantum Requirements Engineering. In Proceedings of the 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), Hannover, Germany, 4–5 September 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 371–374. [ Google Scholar ] [ CrossRef ]
• Upama, P.B.; Faruk, M.J.H.; Nazim, M.; Masum, M.; Shahriar, H.; Uddin, G.; Barzanjeh, S.; Ahamed, S.I.; Rahman, A. Evolution of quantum computing: A systematic survey on the use of quantum computing tools. In Proceedings of the 2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC), Los Alamitos, CA, USA, 27 June–1 July 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 520–529. [ Google Scholar ]
• Gill, S.S.; Kumar, A.; Singh, H.; Singh, M.; Kaur, K.; Usman, M.; Buyya, R. Quantum computing: A taxonomy, systematic review and future directions. Softw. Pract. Exp. 2022 , 52 , 66–114. [ Google Scholar ] [ CrossRef ]
• Almudever, C.G.; Lao, L.; Fu, X.; Khammassi, N.; Ashraf, I.; Iorga, D.; Varsamopoulos, S.; Eichler, C.; Wallraff, A.; Geck, L.; et al. The engineering challenges in quantum computing. In Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE), Lausanne, Switzerland, 27–31 March 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 836–845. [ Google Scholar ]
• Khan, A.A.; Ahmad, A.; Waseem, M.; Liang, P.; Fahmideh, M.; Mikkonen, T.; Abrahamsson, P. Software architecture for quantum computing systems—A systematic review. J. Syst. Softw. 2023 , 201 , 111682. [ Google Scholar ] [ CrossRef ]
• Kitchenham, B.; Charters, S. Guidelines for Performing Systematic Literature Reviews in Software Engineering ; Technical Report; Software Engineering Group School of Computer Science and Mathematics Keele University Keele: Staffs, UK, 2007. [ Google Scholar ]
• Brereton, P.; Kitchenham, B.A.; Budgen, D.; Turner, M.; Khalil, M. Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 2007 , 80 , 571–583. [ Google Scholar ] [ CrossRef ]
• Sepúlveda, S.; Cravero, A. Systematic Review Protocol: Requirements Engineering in Quantum Computing. arXiv 2023 , arXiv:2311.12998. [ Google Scholar ]
• Petticrew, M.; Roberts, H. Systematic Reviews in the Social Sciences: A Practical Guide ; John Wiley & Sons: Hoboken, NJ, USA, 2008. [ Google Scholar ]
• Chadegani, A.; Salehi, H.; Yunus, M.; Farhadi, H.; Fooladi, M.; Farhadi, M.; Ebrahim, N.A. A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases. Political Econ.—Dev. Public Serv. Deliv. 2013. [ Google Scholar ] [ CrossRef ]
• Zhu, J.; Liu, W. A tale of two databases: The use of Web of Science and Scopus in academic papers. Scientometrics 2020 , 123 , 321–335. [ Google Scholar ] [ CrossRef ]
• Wohlin, C. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international Conference on Evaluation and Assessment in Software Engineering, London, UK, 13–14 May 2014; pp. 1–10. [ Google Scholar ]
• Gwet, K. Inter-rater reliability: Dependency on trait prevalence and marginal homogeneity. Stat. Methods Inter-Rater Reliab. Assess. Ser. 2002 , 2 , 9. [ Google Scholar ]
• Fleiss, J.L.; Levin, B.; Paik, M.C. Statistical Methods for Rates and Proportions ; John Wiley & Sons: Hoboken, NJ, USA, 2013. [ Google Scholar ]
• ISO/IEC 9126:2001 ; Software Engineering—Product Quality. International Organization for Standardization: Geneva, Switzerland, 2001.
• Yang, L.; Zhang, H.; Shen, H.; Huang, X.; Zhou, X.; Rong, G.; Shao, D. Quality assessment in systematic literature reviews: A software engineering perspective. Inf. Softw. Technol. 2021 , 130 , 106397. [ Google Scholar ] [ CrossRef ]
• Kitchenham, B.A.; Sjøberg, D.I.; Dybå, T.; Pfahl, D.; Brereton, P.; Budgen, D.; Höst, M.; Runeson, P. Three empirical studies on the agreement of reviewers about the quality of software engineering experiments. Inf. Softw. Technol. 2012 , 54 , 804–819. [ Google Scholar ] [ CrossRef ]
• Budgen, D.; Brereton, P. Performing systematic literature reviews in software engineering. In Proceedings of the 28th International Conference on Software Engineering, Shanghai, China, 20–28 May 2006; pp. 1051–1052. [ Google Scholar ]
• Dybå, T.; Dingsøyr, T. Empirical studies of agile software development: A systematic review. Inf. Softw. Technol. 2008 , 50 , 833–859. [ Google Scholar ] [ CrossRef ]
• Zhou, Y.; Zhang, H.; Huang, X.; Yang, S.; Babar, M.A.; Tang, H. Quality assessment of systematic reviews in software engineering: A tertiary study. In Proceedings of the EASE ’15: 19th International Conference on Evaluation and Assessment in Software Engineering, Nanjing, China, 27–29 April 2015. [ Google Scholar ] [ CrossRef ]
• Sepúlveda, S.; Cravero, A.; Cachero, C. Requirements modeling languages for software product lines: A systematic literature review. Inf. Softw. Technol. 2016 , 69 , 16–36. [ Google Scholar ] [ CrossRef ]
• Usman, M.; Bin Ali, N.; Wohlin, C. A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering. arXiv 2021 , arXiv:2109.10134. [ Google Scholar ] [ CrossRef ]
• Dybå, T.; Dingsøyr, T. Strength of evidence in systematic reviews in software engineering. In Proceedings of the ESEM ’08: Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, Lund, Sweden, 19–20 September 2008; pp. 178–187. [ Google Scholar ] [ CrossRef ]
• Chen, Y.; Zhang, Q.; Peng, W.; Liu, D.; You, Y.; Liu, X.; Tang, S.; Zhang, T.E. Efficacy and safety of mesenchymal stem cells for the treatment of patients infected with COVID-19: A systematic review and meta-analysis protocol. BMJ Open 2020 , 10 , e042085. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Darley, S.; Coulson, T.; Peek, N.; Moschogianis, S.; van der Veer, S.N.; Wong, D.C.; Brown, B. Understanding How the Design and Implementation of Online Consultations Affect Primary Care Quality: Systematic Review of Evidence With Recommendations for Designers, Providers, and Researchers. J. Med. Internet Res. 2022 , 24 , e37436. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Filho, C.A.T.T.; Junior, E.P.; Vendrame, J.W.; Silva, G.M.D.; Cavina, A.P.S.; Lemos, L.K.; Vanderlei, F. Effect of aerobic training with blood flow restriction on strength and hypertrophy: A meta-analysis. Int. J. Sport. Med. 2024 . [ Google Scholar ] [ CrossRef ]
• Gagliardoni, T. Quantum Security of Cryptographic Primitives. arXiv 2017 , arXiv:1705.02417. [ Google Scholar ]
• V, M.C.; N, N.A. A Hybrid Double Encryption Approach for Enhanced Cloud Data Security in Post-Quantum Cryptography. Int. J. Adv. Comput. Sci. Appl. 2023 , 14 , 242–248. [ Google Scholar ] [ CrossRef ]
• Takagi, T.; Peyrin, T. Advances in Cryptology–ASIACRYPT 2017: 23rd International Conference on the Theory and Applications of Cryptology and Information Security, Hong Kong, China, 3–7 December 2017; Proceedings, Part II. Springer: Berlin/Heidelberg, Germany, 2017; Volume 10625.
• Paul, S.; Scheible, P. Towards Post-Quantum Security for Cyber-Physical Systems: Integrating PQC into Industrial M2M Communication. IACR Cryptol. EPrint Arch. 2020 , 2021 , 1563. [ Google Scholar ] [ CrossRef ]
• Frantzi, K.; Ananiadou, S.; Mima, H. Automatic recognition of multi-word terms:. the c-value/nc-value method. Int. J. Digit. Libr. 2000 , 3 , 115–130. [ Google Scholar ] [ CrossRef ]
• Zhou, X.; Jin, Y.; Zhang, H.; Li, S.; Huang, X. A map of threats to validity of systematic literature reviews in software engineering. In Proceedings of the 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), Hamilton, New Zealand, 6–9 December 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 153–160. [ Google Scholar ]
• Ampatzoglou, A.; Bibi, S.; Avgeriou, P.; Verbeek, M.; Chatzigeorgiou, A. Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf. Softw. Technol. 2019 , 106 , 201–230. [ Google Scholar ] [ CrossRef ]
• Petersen, K.; Gencel, C. Worldviews, research methods, and their relationship to validity in empirical software engineering research. In Proceedings of the 2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement, Ankara, Turkey, 23–26 October 2013; IEEE: Piscataway, NJ, USA, 2013; pp. 81–89. [ Google Scholar ]
• Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021 , 372 , n71. [ Google Scholar ] [ CrossRef ] [ PubMed ]

Click here to enlarge figure

Share and Cite

Sepúlveda, S.; Cravero, A.; Fonseca, G.; Antonelli, L. Systematic Review on Requirements Engineering in Quantum Computing: Insights and Future Directions. Electronics 2024 , 13 , 2989. https://doi.org/10.3390/electronics13152989

Sepúlveda S, Cravero A, Fonseca G, Antonelli L. Systematic Review on Requirements Engineering in Quantum Computing: Insights and Future Directions. Electronics . 2024; 13(15):2989. https://doi.org/10.3390/electronics13152989

Sepúlveda, Samuel, Ania Cravero, Guillermo Fonseca, and Leandro Antonelli. 2024. "Systematic Review on Requirements Engineering in Quantum Computing: Insights and Future Directions" Electronics 13, no. 15: 2989. https://doi.org/10.3390/electronics13152989

Article Metrics

Article access statistics, supplementary material.

ZIP-Document (ZIP, 98 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Tools to support systematic reviews in software engineering: a cross-domain survey using semi-structured interviews

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations.

• Pauzi Z Capiluppi A (2024) Beyond the Systematic: Forecasting Importance and Emergence of Research Areas in Applications of Software Traceability Using NLP Evaluation of Novel Approaches to Software Engineering 10.1007/978-3-031-64182-4_6 (119-140) Online publication date: 10-Jul-2024 https://doi.org/10.1007/978-3-031-64182-4_6
• Faroun H Zary N Baqer K Alkhaja F Gad K Alameddine M Al Suwaidi H (2023) Identification of Key Factors for Optimized Health Care Services: Protocol for a Multiphase Study of the Dubai Vaccination Campaign JMIR Research Protocols 10.2196/42278 12 (e42278) Online publication date: 17-Apr-2023 https://doi.org/10.2196/42278
• Elfadil N Ibrahim I (2022) Embedded System Design Student’s Learning Readiness Instruments: Systematic Literature Review Frontiers in Education 10.3389/feduc.2022.799683 7 Online publication date: 18-Feb-2022 https://doi.org/10.3389/feduc.2022.799683
• Show More Cited By

Index Terms

Software and its engineering

Recommendations

Systematic review in software engineering: where we are and where we should be going.

In 2004 Kitchenham et al. first proposed the idea of evidence-based software engineering (EBSE). EBSE requires a systematic and unbiased method of aggregating empirical studies and has encouraged software engineering researches to undertake systematic ...

Systematic Literature Reviews in Distributed Software Development: A Tertiary Study

Distributed Software Development (DSD) emerged from the need to achieve geographically distant customers and currently, allows organizations have global customers and other benefits. This scenario has given rise to new Software Engineering challenges ...

The use of systematic reviews in evidence based software engineering: a systematic mapping study

Context . A decade ago, Kitchenham, Dybå and Jørgensen argued that software engineering could benefit from an evidence-based research approach similar that that used in medicine, introducing the basis for Evidence Based Software Engineering (EBSE). ...

Information

Published in.

• General Chair:

Nanjing University, China

• Program Chairs:

University of Adelaide, Australia

• NJU: Nanjing University

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates, author tags.

• automated tools
• systematic review
• Research-article

Acceptance Rates

Contributors, other metrics, bibliometrics, article metrics.

• 26 Total Citations View Citations
• 953 Total Downloads
• Downloads (Last 12 months) 122
• Downloads (Last 6 weeks) 5
• YÜNKÜL E (2022) Always with me even from a distance: Teachers' technology usage skills before and after distance education Journal of Educational Technology and Online Learning 10.31681/jetol.1118048 5 :3 (636-654) Online publication date: 30-Sep-2022 https://doi.org/10.31681/jetol.1118048
• Cierco Jimenez R Lee T Rosillo N Cordova R Cree I Gonzalez A Indave Ruiz B (2022) Machine learning computational tools to assist the performance of systematic reviews: A mapping review BMC Medical Research Methodology 10.1186/s12874-022-01805-4 22 :1 Online publication date: 16-Dec-2022 https://doi.org/10.1186/s12874-022-01805-4
• Santoso N Sulistyaningtyas I Pratama B (2022) Transformational Leadership During the COVID-19 Pandemic: Strengthening Employee Engagement Through Internal Communication Journal of Communication Inquiry 10.1177/01968599221095182 (019685992210951) Online publication date: 18-Apr-2022 https://doi.org/10.1177/01968599221095182
• Felizardo K Ramos A O. Melo C de Souza É Vijaykumar N Nakagawa E (2021) Global and Latin American female participation in evidence-based software engineering: a systematic mapping study Journal of the Brazilian Computer Society 10.1186/s13173-021-00109-7 27 :1 Online publication date: 19-Mar-2021 https://doi.org/10.1186/s13173-021-00109-7
• Napoleao B Petrillo F Halle S (2021) Automated Support for Searching and Selecting Evidence in Software Engineering: A Cross-domain Systematic Mapping 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 10.1109/SEAA53835.2021.00015 (45-53) Online publication date: Sep-2021 https://doi.org/10.1109/SEAA53835.2021.00015
• Groppe S Hartung L (2020) ReViz: A Tool for Automatically Generating Citation Graphs and Variants Digital Libraries at Times of Massive Societal Transition 10.1007/978-3-030-64452-9_10 (107-121) Online publication date: 26-Nov-2020 https://doi.org/10.1007/978-3-030-64452-9_10
• Canedo E Parente da Costa R Vieira Amaral L Coutinho M Daniel Amvame Nze G de Sousa Junior R (2019) Proposal of an Implementation Methodology of ICT Processes Information 10.3390/info10110327 10 :11 (327) Online publication date: 23-Oct-2019 https://doi.org/10.3390/info10110327

View Options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

View options.

View or Download as a PDF file.

View online with eReader .

Share this Publication link

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw

• Journal of Systems and Software 80(4):571-583
• 80(4):571-583
• CC BY-NC-ND 3.0
• This person is not on ResearchGate, or hasn't claimed this research yet.

• Keele University

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Guillermo Fonseca
• Leandro Antonelli

• J POWER SOURCES

• Shikha Saloni

• Bhaskara Srinivas
• Sumit Kumar
• Prachi Ahlawat
• Jyoti Sahni

• Ws Richardson
• W. H. Rosenberg

• David Budgen

• Richard P. Bagozzi
• Paul R. Warshaw
• INFORM SOFTWARE TECH
• Martin Shepperd
• Mark Staples

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

Doing a Systematic Review: A Student’s Guide

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

What is Systematic Review?

A systematic review is a comprehensive, structured analysis of existing research on a specific topic. It uses predefined criteria to identify, evaluate, and synthesize relevant studies, aiming to provide an unbiased summary of the current evidence.

The explicit and systematic approach of a systematic review distinguishes it from traditional reviews and commentaries.

Here are some key ways that systematic reviews differ from narrative reviews:

• Goals: Narrative reviews provide a summary or overview of a topic, while systematic reviews answer a focused review question.
• Sources of Literature: Narrative reviews often use a non-exhaustive and unstated body of literature, which can lead to publication bias. Systematic reviews consider a list of databases, grey literature, and other sources.
• Selection Criteria: Narrative reviews usually use subjective or no selection criteria, which can lead to selection bias. Systematic reviews have a clear and explicit selection process.
• Appraisal of Study Quality: Narrative reviews vary in their evaluation of study quality. Systematic reviews use standard checklists for a rigorous appraisal of study quality.

Systematic reviews are time-intensive and need a research team with multiple skills and contributions. There are some cases where systematic reviews are unable to meet the necessary objectives of the review question.

In these cases, scoping reviews (which are sometimes called scoping exercises/scoping studies) may be more useful to consider.

Scoping reviews are different from systematic reviews because they may not include a mandatory critical appraisal of the included studies or synthesize the findings from individual studies.

Assessing The Need For A Systematic Review

When assessing the need for a systematic review, one must first check if any existing or ongoing reviews already exist and determine if a new review is justified.

Scoping reviews frequently serve as preliminary steps before conducting full systematic reviews. They help assess the available literature’s breadth, identify key concepts, and determine the feasibility of a more comprehensive review.

This initial exploration guides researchers in refining their approach for subsequent in-depth analyses.

This process should begin by searching relevant databases.

Resources to consider searching include:

• NICE : National Institute for Health and Clinical Excellence
• Campbell Library of Systematic Reviews for reviews in education, crime and justice, and social welfare
• EPPI : Evidence for Policy and Practice Information Centre, particularly their database of systematic and non-systematic reviews of public health interventions (DoPHER)
• MEDLINE : Primarily covers the medical domain, making it a primary resource for systematic reviews concerning healthcare interventions
• PsycINFO : For research in psychology, psychiatry, behavioral sciences, and social sciences
• Cochrane Library (specifically CDSR) : Focuses on systematic reviews of health care interventions, providing regularly updated and critically appraised reviews

If an existing review addressing the question of interest is found, its quality should be assessed to determine its suitability for guiding policy and practice.

If a high-quality, relevant review is located, but its completion date is some time ago, updating the review might be warranted.

Assessing current relevance is vital, especially in rapidly evolving research fields. Collaboration with the original research team might be beneficial during the update process, as they could provide access to their data.

If the review is deemed to be of adequate quality and remains relevant, undertaking another systematic review may not be necessary.

When a new systematic review or an update is deemed necessary, the subsequent step involves establishing a review team and potentially an advisory group, who will then develop the review protocol.

How To Conduct A Systematic Review

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is a reporting guideline designed to improve the transparency and completeness of systematic review reporting.

PRISMA was created to tackle the issue of inadequate reporting often found in systematic reviews:

• Checklist : PRISMA features a 27-item checklist covering all aspects of a systematic review, from the rationale and objectives to the synthesis of findings and discussion of limitations. Each checklist item is accompanied by detailed reporting recommendations in an Explanation and Elaboration document .
• Flow Diagram : PRISMA also includes a flow diagram to visually represent the study selection process, offering a clear, standardized way to illustrate how researchers arrived at the final set of included studies.

Step 1: write a research protocol

A protocol in the context of systematic reviews is a detailed plan that outlines the methodology to be employed throughout the review process.

The protocol serves as a roadmap, guiding researchers through each stage of the review in a transparent and replicable manner.

This document should provide specific details about every stage of the research process, including the methodology for identifying, selecting, and analyzing relevant studies.

For example, the protocol should specify search strategies for relevant studies, including whether the search will encompass unpublished works.

The protocol should be created before beginning the research process to ensure transparency and reproducibility.

This pre-determined plan ensures that decisions made during the review are objective and free from bias, as they are based on pre-established criteria.

Protocol modifications are sometimes necessary during systematic reviews. While adhering to the protocol is crucial for minimizing bias, there are instances where modifications are justified. For instance, a deeper understanding of the research question that emerges from examining primary research might necessitate changes to the protocol.

Systematic reviews should be registered at inception (at the protocol stage) for these reasons:

• To help avoid unplanned duplication
• To enable the comparison of reported review methods with what was planned in the protocol

This registration prevents duplication (research waste) and makes the process easy when the full systematic review is sent for publication.

PROSPERO is an international database of prospectively registered systematic reviews in health and social care. Non-Cochrane protocols should be registered on PROSPERO.

Research Protocol

Rasika Jayasekara, Nicholas Procter. The effects of cognitive behaviour therapy for major depression in older adults: a systematic review. PROSPERO 2012 CRD42012003151 Available from:  https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42012003151

Review question

How effective is CBT compared with other interventions, placebo or standard treatment in achieving relapse prevention and improving mental status for older adults with major depression?

The search strategy aims to find both published and unpublished studies and publications. The search will be limited to English language papers published from 2002 to 2012.

A three-step search strategy will be developed using MeSH terminology and keywords to ensure that all materials relevant to the review are captured.

An initial limited search of MEDLINE and CINAHL will be undertaken followed by an analysis of the text words contained in the title and abstract, and of the index terms used to describe the article. A second search using all identified keywords and index terms will then be undertaken.

Thirdly, the reference list of all identified reports and articles will be searched for additional studies.

The databases to be searched included:

• Cochrane Central Register of Controlled Trials
• Controlled Trials
• Current Contents

The search for unpublished studies will include:

• Digital Dissertations (Proquest)
• Conference Proceedings

Experts in the field will be contacted for ongoing and unpublished trials. Experts will be identified through journal publications.

Types of study to be included

All randomised controlled trials (RCTs) assessing the effectiveness of CBT as a treatment for older adults with major depression when compared to standard care, specific medication, other therapies and no intervention will be considered.

In the absence of RCTs, other research designs such as quasi-experimental studies, case-controlled studies and cohort studies will be examined. However, descriptive studies and expert opinion will be excluded.

Condition or domain being studied

Major depression is diagnosed according to DSM IV or ICD 10 criteria.

Where trials fail to employ diagnostic criteria, the severity of depression will be described by the use of standardised rating scales, including the Hamilton Depression Rating Scale, Montgomery and Asberg Rating Scale and the Geriatric Depression Rating Scale.

The trials including participants with an explicit diagnosis of dementia or Parkinson’s disease and other mental illnesses will be excluded.

The review will include trials conducted in primary, secondary, community, nursing homes and in-patient settings.

Participants/population

The review will include trials in which patients are described as elderly, geriatric, or older adults, or in which all patients will be aged 55 or over (many North American trials of older adult populations use a cut-off of 55 years).

The review will include trials with subjects of either sex. Where possible, participants will be categorised as community or long term care residents.

Intervention(s), exposure(s)

The review will focus on interventions designed to assess the effects of CBT for older adults with major depression.

The label cognitive behavioural therapy has been applied to a variety of interventions and, accordingly, it is difficult to provide a single, unambiguous definition.

In order to be classified as CBT the intervention must clearly demonstrate the following components:

• the intervention involves the recipient establishing links between their thoughts, feelings and actions with respect to the target symptom;
• the intervention involves the correction of the person’s misperceptions, irrational beliefs and reasoning biases related to the target symptom.
• – the recipient monitoring his or her own thoughts, feelings and behaviours with respect to the target symptom; and
• – the promotion of alternative ways of coping with the target symptom.

In addition, all therapies that do not meet these criteria (or that provide insufficient information) but are labelled as ‘CBT’ or ‘Cognitive Therapy’ will be included as ‘less well defined’ CBT.

Comparator(s)/control

other interventions, placebo or standard treatment

Main outcome(s)

Primary outcomes

• Depression level as assessed by Hamilton Depression Rating Scale, Montgomery or Asberg Rating Scale or the Geriatric Depression Rating Scale.
• Relapse (as defined in the individual studies)
• Death (sudden, unexpected death or suicide).
• Psychological well being (as defined in the individual studies)

Measures of effect

The review will categorise outcomes into those measured in the shorter term (within 12 weeks of the onset of therapy), medium term (within 13 to 26 weeks of the onset of therapy) and longer term (over 26 weeks since the onset of therapy).

Additional outcome(s)

Secondary outcomes

• Mental state
• Quality of life
• Social functioning
• Hospital readmission
• Unexpected or unwanted effect (adverse effects), such as anxiety, depression and dependence on the relationship with the therapist

Data extraction (selection and coding)

Data will be extracted from papers included in the review using JBI-MAStARI. In this stage, any relevant studies will be extracted in relation to their population, interventions, study methods and outcomes.

Where data are missing or unclear, authors will be contacted to obtain information.

Risk of bias (quality) assessment

All papers selected for retrieval will be assessed by two independent reviewers for methodological validity prior to inclusion in the review.

Since the review will evaluate the experimental studies only, The Joanna Briggs Institute Meta Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI) will be used to evaluate each study’s methodological validity.

If there is a disagreement between the two reviewers, there will be a discussion with the third reviewer to solve the dissimilarity.

Strategy for data synthesis

Where possible quantitative research study results will be pooled in statistical meta-analysis using Review Manager Software from the Cochrane Collaboration.

Odds ratio (for categorical outcome data) or standardised mean differences (for continuous data) and their 95% confidence intervals will be calculated for each study.

Heterogeneity will be assessed using the standard Chi-square. Where statistical pooling is not possible the findings will be presented in narrative form.

Step 2: formulate a research question 

Developing a focused research question is crucial for a systematic review, as it underpins every stage of the review process.

The question defines the review’s nature and scope, guides the identification of relevant studies, and shapes the data extraction and synthesis processes.

It’s essential that the research question is answerable and clearly stated in the review protocol, ensuring that the review’s boundaries are well-defined.

A narrow question may limit the number of relevant studies and generalizability, while a broad question can make it challenging to reach specific conclusions.

PICO Framework

The PICO framework is a model for creating focused clinical research questions. The acronym PICO stands for:

• P opulation/Patient/Problem: This element defines the specific group of people the research question pertains to.
• I ntervention: This is the treatment, test, or exposure being considered for the population.
• C omparison: This is the alternative intervention or control group against which the intervention is being compared.
• O utcome: This element specifies the results or effects of the interventions being investigated

Using the PICO format when designing research helps to minimize bias because the questions and methods of the review are formulated before reviewing any literature.

The PICO elements are also helpful in defining the inclusion criteria used to select sources for the systematic review.

The PICO framework is commonly employed in systematic reviews that primarily analyze data from randomized controlled trials .

Not every element of PICO is required for every research question. For instance, it is not always necessary to have a comparison

Types of questions that can be answered using PICO:

“In patients with a recent acute stroke (less than 6 weeks) with reduced mobility ( P ), is any specific physiotherapy approach ( I ) more beneficial than no physiotherapy ( C ) at improving independence in activities of daily living and gait speed ( O )?

“For women who have experienced domestic violence ( P ), how effective are advocacy programmes ( I ) compared to other treatments ( C ) on improving the quality of life ( O )?”

Etiology/Harm

Are women with a history of pelvic inflammatory disease (PID) ( P ) at higher risk for gynecological cancers ( O ) than women with no history of PID ( C )?

Among asymptomatic adults at low risk of colon cancer ( P ), is fecal immunochemical testing (FIT) ( I ) as sensitive and specific for diagnosing colon cancer ( O ) as colonoscopy ( C )?

Among adults with pneumonia ( P ), do those with chronic kidney disease (CKD) ( I ) have a higher mortality rate ( O ) than those without CKD ( C )?

Alternative Frameworks

• PICOCS : This framework, used in public health research, adds a “ C ontext” element to the PICO framework. This is useful for examining how the environment or setting in which an intervention is delivered might influence its effectiveness.
• PICOC : This framework expands on PICO by incorporating “ C osts” as an element of the research question. It is particularly relevant to research questions involving economic evaluations of interventions.
• ECLIPSE : E xpectations, C lient group, L ocation, I mpact, P rofessionals involved, S ervice, and E valuation. It is a mnemonic device designed to aid in searching for health policy and management information.
• PEO : This acronym, standing for P atient, E xposure, and O utcome, is a variation of PICO used when the research question focuses on the relationship between exposure to a risk factor and a specific outcome.
• PIRD : This acronym stands for P opulation, I ndex Test, R eference Test, and D iagnosis of Interest, guiding research questions that focus on evaluating the diagnostic accuracy of a particular test.
• PFO : This acronym, representing P opulation, P rognostic F actors, and O utcome, is tailored for research questions that aim to investigate the relationship between specific prognostic factors and a particular health outcome.
• SDMO : This framework, which stands for S tudies, D ata, M ethods, and O utcomes, assists in structuring research questions focused on methodological aspects of research, examining the impact of different research methods or designs on the quality of research findings.

Step 3: Search Strategy

PRISMA  (Preferred Reporting Items for Systematic reviews and Meta-Analyses) provide appropriate guidance for reporting quantitative literature searches.

Present the full search strategies for all databases, registers and websites, including any filters and limits used. PRISMA 2020 Checklist

A search strategy is a comprehensive and reproducible plan for identifying all relevant research studies that address a specific research question.

This systematic approach to searching helps minimize bias and distinguishes systematic reviews from other types of literature reviews.

It’s important to be transparent about the search strategy and document all decisions for auditability. The goal is to identify all potentially relevant studies for consideration.

Here’s a breakdown of a search strategy:

Search String Construction

It is recommended to consult topic experts on the review team and advisory board in order to create as complete a list of search terms as possible for each concept.

To retrieve the most relevant results, a search string is used. This string is made up of:

• Keywords:  Search terms should be relevant to the subject areas of the research question and should be identified for all components of the research question (e.g., Population, Intervention, Comparator, and Outcomes – PICO). Using relevant keywords helps minimize irrelevant search returns. Sources such as dictionaries, textbooks, and published articles can help identify appropriate keywords.
• Synonyms: These are words or phrases with similar meanings to the keywords, as authors may use different terms to describe the same concepts. Including synonyms helps cover variations in terminology and increases the chances of finding all relevant studies. For example, a drug intervention may be referred to by its generic name or by one of its several proprietary names.
• Truncation symbols : These broaden the search by capturing variations of a keyword. They function by locating every word that begins with a specific root. For example, if a user was researching interventions for smoking, they might use a truncation symbol to search for “smok*” to retrieve records with the words “smoke,” “smoker,” “smoking,” or “smokes.” This can save time and effort by eliminating the need to input every variation of a word into a database.
• Boolean operators: The use of Boolean operators (AND/OR/NEAR/NOT) helps to combine these terms effectively, ensuring that the search strategy is both sensitive and specific. For instance, using “AND” narrows the search to include only results containing both terms, while “OR” expands it to include results containing either term.

Information Sources

The primary goal is to find all published and unpublished studies that meet the predefined criteria of the research question. This includes considering various sources beyond typical databases

Information sources for systematic reviews can include a wide range of resources like scholarly databases, unpublished literature, conference papers, books, and even expert consultations.

Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted. PRISMA 2020 Checklist

An exhaustive, systematic search strategy is developed with the assistance of an expert librarian.

• Electronic Databases : Searches should include seven key databases: CINAHL, Medline, APA PsycArticles, Psychology and Behavioral Sciences Collection, APA PsycInfo, SocINDEX with Full Text, and Web of Science: Core Collections.
• Grey Literature : In addition to databases, forensic or ‘expansive’ searches can be conducted. This includes: grey literature database searches (e.g.  OpenGrey , WorldCat , Ethos ),  conference proceedings, unpublished reports, theses  , clinical trial databases , searches by names of authors of relevant publications. Independent research bodies may also be good sources of material, e.g. Centre for Research in Ethnic Relations , Joseph Rowntree Foundation , Carers UK .
• Citation Searching : Reference lists often lead to highly cited and influential papers in the field, providing valuable context and background information for the review.
• Handsearching : Manually searching through specific journals or conference proceedings page-by-page is another way to ensure all relevant studies are captured, particularly those not yet indexed in databases.
• Contacting Experts : Reaching out to researchers or experts in the field can provide access to unpublished data or ongoing research not yet publicly available.

It is important to note that this may not be an exhaustive list of all potential databases.

A systematic computerized search was performed for publications that appeared between 1974 and 2018 in English language journals. Four databases were searched including PsychINFO, Embase, OVOID MEDLINE, and AMED. The databases were searched with combinations of search terms relating to attachment (“attachment” OR “working model” OR “safe haven” OR “secure base” OR “felt security”) AND romantic couples (“dyad” OR “couple” OR “spous” OR “partner” OR “romantic” OR “wife” OR “husband” OR “close relationship” OR “interpersonal” OR “intimate” OR “mari”) AND social support (“support prov” OR “caregiving” OR “support giv” OR “social support” OR “enacted support” OR “support received” OR “receiv* support” OR “prov support” OR “dyadic coping” OR “interpersonal coping” OR “collaborative coping” OR “help‐seeking” OR “emotional support” OR “tangible support” OR “instrumental support” OR “perceived support” OR “responsive” OR “buffer” OR “partner support” OR “Support avail*” OR “available support”). The reference lists of the retrieved studies were checked to find other relevant publications, which were not identified in the computerized database searches.

Inclusion Criteria

Specify the inclusion and exclusion criteria for the review. PRISMA 2020 Checklist

Before beginning the literature search, researchers should establish clear eligibility criteria for study inclusion.

Inclusion criteria are used to select studies for a systematic review and should be based on the study’s research method and PICO elements.

To maintain transparency and minimize bias, eligibility criteria for study inclusion should be established a priori. Ideally, researchers should aim to include only high-quality randomized controlled trials that adhere to the intention-to-treat principle.

The selection of studies should not be arbitrary, and the rationale behind inclusion and exclusion criteria should be clearly articulated in the research protocol.

When specifying the inclusion and exclusion criteria, consider the following aspects:

• Intervention Characteristics: Researchers might decide that, in order to be included in the review, an intervention must have specific characteristics. They might require the intervention to last for a certain length of time, or they might determine that only interventions with a specific theoretical basis are appropriate for their review.
• Population Characteristics: A systematic review might focus on the effects of an intervention for a specific population. For instance, researchers might choose to focus on studies that included only nurses or physicians.
• Outcome Measures: Researchers might choose to include only studies that used outcome measures that met a specific standard.
• Age of Participants: If a systematic review is examining the effects of a treatment or intervention for children, the authors of the review will likely choose to exclude any studies that did not include children in the target age range.
• Diagnostic Status of Participants: Researchers conducting a systematic review of treatments for anxiety will likely exclude any studies where the participants were not diagnosed with an anxiety disorder.
• Study Design: Researchers might determine that only studies that used a particular research design, such as a randomized controlled trial, will be included in the review.
• Control Group: In a systematic review of an intervention, researchers might choose to include only studies that included certain types of control groups, such as a waiting list control or another type of intervention.
• Publication status : Decide whether only published studies will be included or if unpublished works, such as dissertations or conference proceedings, will also be considered.

Studies that met the following criteria were included: (a) empirical studies of couples (of any gender) who are in a committed romantic relationship, whether married or not; (b) measurement of the association between adult attachment and support in the context of this relationship; (c) the article was a full report published in English; and (d) the articles were reports of empirical studies published in peer‐reviewed journals, dissertations, review papers, and conference presentations.

Iterative Process

The iterative nature of developing a search strategy for systematic reviews stems from the need to refine and adapt the search process based on the information encountered at each stage.

A single attempt rarely yields the perfect final strategy. Instead, it is an evolving process involving a series of test searches, analysis of results, and discussions among the review team.

Here’s how the iterative process unfolds:

• Initial Strategy Formulation: Based on the research question, the team develops a preliminary search strategy, including identifying relevant keywords, synonyms, databases, and search limits.
• Test Searches and Refinement: The initial search strategy is then tested on chosen databases. The results are reviewed for relevance, and the search strategy is refined accordingly. This might involve adding or modifying keywords, adjusting Boolean operators, or reconsidering the databases used.
• Discussions and Iteration: The search results and proposed refinements are discussed within the review team. The team collaboratively decides on the best modifications to improve the search’s comprehensiveness and relevance.
• Repeating the Cycle: This cycle of test searches, analysis, discussions, and refinements is repeated until the team is satisfied with the strategy’s ability to capture all relevant studies while minimizing irrelevant results.

The iterative nature of developing a search strategy is crucial for ensuring that the systematic review is comprehensive and unbiased.

By constantly refining the search strategy based on the results and feedback, researchers can be more confident that they have identified all relevant studies.

This iterative process ensures that the applied search strategy is sensitive enough to capture all relevant studies while maintaining a manageable scope.

Throughout this process, meticulous documentation of the search strategy, including any modifications, is crucial for transparency and future replication of the systematic review.

Step 4: Search the Literature

Conduct a systematic search of the literature using clearly defined search terms and databases.

Applying the search strategy involves entering the constructed search strings into the respective databases’ search interfaces. These search strings, crafted using Boolean operators, truncation symbols, wildcards, and database-specific syntax, aim to retrieve all potentially relevant studies addressing the research question.

The researcher, during this stage, interacts with the database’s features to refine the search and manage the retrieved results.

This might involve employing search filters provided by the database to focus on specific study designs, publication types, or other relevant parameters.

Applying the search strategy is not merely a mechanical process of inputting terms; it demands a thorough understanding of database functionalities and a discerning eye to adjust the search based on the nature of retrieved results.

Step 5: screening and selecting research articles

Once the search strategy is finalized, it is applied to the selected databases, yielding a set of search results.

These search results are then screened against pre-defined inclusion criteria to determine their eligibility for inclusion in the review.

The goal is to identify studies that are both relevant to the research question and of sufficient quality to contribute to a meaningful synthesis.

Studies meeting the inclusion criteria are usually saved into electronic databases, such as Endnote or Mendeley , and include title, authors, date and publication journal along with an abstract (if available).

Study Selection

Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process. PRISMA 2020 Checklist

The selection process in a systematic review involves multiple reviewers to ensure rigor and reliability.

To minimize bias and enhance the reliability of the study selection process, it is recommended that at least two reviewers independently assess the eligibility of each study. This independent assessment helps reduce the impact of individual biases or errors in judgment.

• Initial screening of titles and abstracts: After applying a strategy to search the literature, the next step involves screening the titles and abstracts of the identified articles against the predefined inclusion and exclusion criteria. During this initial screening, reviewers aim to identify potentially relevant studies while excluding those clearly outside the scope of the review. It is crucial to prioritize over-inclusion at this stage, meaning that reviewers should err on the side of keeping studies even if there is uncertainty about their relevance. This cautious approach helps minimize the risk of inadvertently excluding potentially valuable studies.
• Retrieving and assessing full texts: For studies which a definitive decision cannot be made based on the title and abstract alone, reviewers need to obtain the full text of the articles for a comprehensive assessment against the predefined inclusion and exclusion criteria. This stage involves meticulously reviewing the full text of each potentially relevant study to determine its eligibility definitively.
• Resolution of disagreements : In cases of disagreement between reviewers regarding a study’s eligibility, a predefined strategy involving consensus-building discussions or arbitration by a third reviewer should be in place to reach a final decision. This collaborative approach ensures a fair and impartial selection process, further strengthening the review’s reliability.

First, the search results from separate databases were combined, and any duplicates were removed. The lead author (S. M.) and a postgraduate researcher (F. N.) applied the described inclusion criteria in a standardized manner. First, both the titles and abstracts of the articles were evaluated for relevance. If, on the basis of the title and/or abstract, the study looked likely to meet inclusion criteria hard copies of the manuscripts were obtained. If there was doubt about the suitability of an article, then the manuscript was included in the next step. The remaining articles were obtained for full‐text review, and the method and results sections were read to examine whether the article fitted the inclusion criteria. If there was doubt about the suitability of the manuscripts during this phase, then this article was discussed with another author (C. H.). Finally, the reference lists of the eligible articles were checked for additional relevant articles not identified during the computerized search. For the selected articles (n = 43), the results regarding the relationship between attachment and support were included in this review (see Figure 1, for PRISMA flowchart).

PRISMA Flowchart

The PRISMA flowchart is a visual representation of the study selection process within a systematic review.

The flowchart illustrates the step-by-step process of screening, filtering, and selecting studies based on predefined inclusion and exclusion criteria.

The flowchart visually depicts the following stages:

• Identification: The initial number of titles and abstracts identified through database searches.
• Screening: The screening process, based on titles and abstracts.
• Eligibility: Full-text copies of the remaining records are retrieved and assessed for eligibility.
• Inclusion: Applying the predefined inclusion criteria resulted in the inclusion of publications that met all the criteria for the review.
• Exclusion: The flowchart details the reasons for excluding the remaining records.

This systematic and transparent approach, as visualized in the PRISMA flowchart, ensures a robust and unbiased selection process, enhancing the reliability of the systematic review’s findings.

The flowchart serves as a visual record of the decisions made during the study selection process, allowing readers to assess the rigor and comprehensiveness of the review.

• How to fill a PRISMA flow diagram

Step 6: Criticallay Appraising the Quality of Included Studies

Quality assessment provides a measure of the strength of the evidence presented in a review.

High-quality studies with rigorous methodologies contribute to a more robust and reliable evidence base, increasing confidence in the review’s conclusions.

Conversely, including low-quality studies with methodological weaknesses can undermine the review’s findings and potentially lead to inaccurate recommendations.

To judge the quality of studies included in a systematic review, standardized instruments, such as checklists and scales, are commonly used. These tools help to ensure a transparent and reproducible assessment process.

The choice of tool should be justified and aligned with the study design and the level of detail required. Using quality scores alone is discouraged; instead, individual aspects of methodological quality should be considered.

Here are some specific tools mentioned in the sources:

• Jadad score
• Cochrane Risk of Bias tool
• Cochrane Effective Practice and Organisation of Care (EPOC) Group Risk of Bias Tool
• Quality Assessment of Diagnostic Accuracy Studies (QUADAS)
• Newcastle – Ottawa Quality Assessment Scale for case-control and cohort studies
• EPHPP Assessment Tool
• Critical Appraisal Skills Programme (CASP) Appraisal Checklist
• Cochrane Public Health Group (CPHG)

The quality of the study was not an inclusion criterion; however, a study quality check was carried out. Two independent reviewers (S. M. and C. H.) rated studies that met the inclusion criteria to determine the strength of the evidence. The Effective Public Health Practice Project Quality Assessment Tool for Quantitative Studies was adapted to assess the methodological quality of each study (Thomas, Ciliska, Dobbins, & Micucci, 2004). The tool was adjusted to include domains relevant to the method of each study. For example, blinding was removed for nonexperimental studies. Following recommendations by Thomas et al. (2004) each domain was rated as either weak (3 points), moderate (2 points), or strong (1 point). The mean score across questions was used as an indicator of overall quality, and studies were assigned an overall quality rating of strong (1.00–1.50), moderate (1.51–2.50),

Evidence Tables

Aspects of the appraisal of studies included in the review should be recorded as evidence tables (NICE 2009): simple text tables where the design and scope of studies are summarised.

The reader of the review can use the evidence tables to check the details, and assess the credibility and generalisability of findings, of particular studies.

Critical appraisal of the quality of included studies may be combined with data extraction tables.

Step 7: extracting data from studies

To effectively extract data from studies that meet your systematic review’s inclusion criteria, you should follow a structured process that ensures accuracy, consistency, and minimizes bias.

1. Develop a data extraction form:

• Design a standardized form (paper or electronic) to guide the data extraction process : This form should be tailored to your specific review question and the types of studies included.
• Pilot test the form : Test the form on a small sample of included studies (e.g., 3-5). Assess for clarity, completeness, and usability. Refine the form based on feedback and initial experiences.
• Reliability : Ensure all team members understand how to use the form consistently.

2. Extract the data:

• General Information: This includes basic bibliographic details (journal, title, author, volume, page numbers), study objective as stated by the authors, study design, and funding source.
• Study Characteristics: Capture details about the study population (demographics, inclusion/exclusion criteria, recruitment procedures), interventions (description, delivery methods), and comparators (description if applicable).
• Outcome Data: Record the results of the intervention and how they were measured, including specific statistics used. Clearly define all outcomes for which data are being extracted.
• Risk of Bias Assessment: Document the methods used to assess the quality of the included studies and any potential sources of bias. This might involve using standardized checklists or scales.
• Additional Information: Depending on your review, you may need to extract data on other variables like adverse effects, economic evaluations, or specific methodological details.

3. Dual independent review:

• Ensure that at least two reviewers independently extract data from each study using the standardized form. Cross-check extracted data for accuracy to minimize bias and helps identify any discrepancies.
• Have a predefined strategy for resolving disagreements: This might involve discussion, consensus, or arbitration by a third reviewer.
• Record the reasons for excluding any studies during the data extraction phase. :This enhances the transparency and reproducibility of your review.
• If necessary, contact study authors to obtain missing or clarify unclear information : This is particularly important for data critical to your review’s outcomes.
• Clearly document your entire data extraction process, including any challenges encountered and decisions made. This enhances the transparency and rigor of your systematic review.

By following these steps, you can effectively extract data from studies that meet your inclusion criteria, forming a solid foundation for the analysis and synthesis phases of your systematic review.

Step 8: synthesize the extracted data

The key element of a systematic review is the synthesis: that is the process that brings together the findings from the set of included studies in order to draw conclusions based on the body of evidence.

Data synthesis in a systematic review involves collating, combining, and summarizing findings from the included studies.

This process aims to provide a reliable and comprehensive answer to the review question by considering the strength of the evidence, examining the consistency of observed effects, and investigating any inconsistencies.

The data synthesis will be presented in the results section of the systematic review.

• Develop a clear text narrative that explains the key findings
• Use a logical heading structure to guide readers through your results synthesis
• Ensure your text narrative addresses the review’s research questions
• Use tables to summarise findings (can be same table as data extraction)

Identifying patterns, trends, and differences across studies

Narrative synthesis uses a textual approach to analyze relationships within and between studies to provide an overall assessment of the evidence’s robustness. All systematic reviews should incorporate elements of narrative synthesis, such as tables and text.

Remember, the goal of a narrative synthesis is to go beyond simply summarizing individual studies. You’re aiming to create a new understanding by integrating and interpreting the available evidence in a systematic and transparent way.

Organize your data:

• Group studies by themes, interventions, or outcomes
• Create summary tables to display key information across studies
• Use visual aids like concept maps to show relationships between studies

Describe the studies:

• Summarize the characteristics of included studies (e.g., designs, sample sizes, settings)
• Highlight similarities and differences across studies
• Discuss the overall quality of the evidence

Develop a preliminary synthesis:

• Start by describing the results of individual studies
• Group similar findings together
• Identify overarching themes or trends

Explore relationships:

• Look for patterns in the data
• Identify factors that might explain differences in results across studies
• Consider how study characteristics relate to outcomes

Address contradictions:

• Consider differences in study populations, interventions, or contexts
• Look at methodological differences that might explain discrepancies
• Consider the implications of inconsistent results
• Don’t ignore conflicting findings
• Discuss possible reasons for contradictions

Avoid vote counting:

• Don’t simply tally positive versus negative results
• Instead, consider the strength and quality of evidence for each finding

Assess the robustness of the synthesis:

• Reflect on the strength of evidence for each finding
• Consider how gaps or limitations in the primary studies affect your conclusions
• Discuss any potential biases in the synthesis process

Step 9: discussion section and conclusion

Summarize key findings:.

• Summarize key findings in relation to your research questions
• Highlight main themes or patterns across studies
• Explain the nuances and complexities in the evidence
• Discuss the overall strength and consistency of the evidence
• This provides a clear takeaway message for readers

Consider study quality and context:

• Assess whether higher quality studies tend to show different results
• Examine if findings differ based on study setting or participant characteristics
• This helps readers weigh the relative importance of conflicting findings

Discuss implications:

• For practice: How might professionals apply these findings?
• For policy: What policy changes might be supported by the evidence?
• Consider both positive and negative implications
• This helps translate your findings into real-world applications

Identify gaps and future research:

• Point out areas where evidence is lacking or inconsistent
• Suggest specific research questions or study designs to address these gaps
• This helps guide future research efforts in the field

State strengths and limitations:

• Discuss the strengths of your review (e.g., comprehensive search, rigorous methodology)
• Acknowledge limitations (e.g., language restrictions, potential for publication bias)
• This balanced approach demonstrates critical thinking and helps readers interpret your findings

Minimizing Bias

To reduce bias in a systematic review, it is crucial to establish a systematic and transparent review process that minimizes bias at every stage. Sources provide insights into strategies and methods to achieve this goal.

• Protocol development and publication: Developing a comprehensive protocol before starting the review is essential. Publishing the protocol in repositories like PROSPERO or Cochrane Library promotes transparency and helps avoid deviations from the planned approach, thereby minimizing the risk of bias.
• Transparent reporting: Adhering to reporting guidelines, such as PRISMA, ensures that all essential aspects of the review are adequately documented, increasing the reader’s confidence in the transparency and completeness of systematic review reporting.
• Dual independent review: Employing two or more reviewers independently at multiple stages of the review process (study selection, data extraction, quality assessment) minimizes bias. Any disagreements between reviewers should be resolved through discussion or by consulting a third reviewer. This approach reduces the impact of individual reviewers’ subjective interpretations or errors.
• Rigorous quality assessment: Assessing the methodological quality of included studies is crucial for minimizing bias in the review findings. Using standardized critical appraisal tools and checklists helps identify potential biases within individual studies, such as selection bias, performance bias, attrition bias, and detection bias.
• Searching beyond published literature: Explore sources of “grey literature” such as conference proceedings, unpublished reports, theses, and ongoing clinical trial databases.
• Contacting experts in the field : Researchers can reach out to authors and investigators to inquire about unpublished or ongoing studies
• Considering language bias : Expanding the search to include studies published in languages other than English can help reduce language bias, although this may increase the complexity and cost of the review.

Reading List

• Galante, J., Galante, I., Bekkers, M. J., & Gallacher, J. (2014). Effect of kindness-based meditation on health and well-being: a systematic review and meta-analysis .  Journal of consulting and clinical psychology ,  82 (6), 1101.
• Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses .  Psychological bulletin ,  143 (6), 565.
• Murray, J., Farrington, D. P., & Sekol, I. (2012). Children’s antisocial behavior, mental health, drug use, and educational performance after parental incarceration: a systematic review and meta-analysis .  Psychological bulletin ,  138 (2), 175.
• Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017). A systematic review of personality trait change through intervention .  Psychological bulletin ,  143 (2), 117.
• Chu, C., Buchman-Schmitt, J. M., Stanley, I. H., Hom, M. A., Tucker, R. P., Hagan, C. R., … & Joiner Jr, T. E. (2017). The interpersonal theory of suicide: A systematic review and meta-analysis of a decade of cross-national research.   Psychological bulletin ,  143 (12), 1313.
• McLeod, S., Berry, K., Hodgson, C., & Wearden, A. (2020). Attachment and social support in romantic dyads: A systematic review .  Journal of clinical psychology ,  76 (1), 59-101.