critical appraisal of systematic literature review

• Search Menu
• Sign in through your institution
• Advance Articles
• Specialty Certificate Examination Cases
• Cover Archive
• Virtual Issues
• Trending Articles
• Author Guidelines
• Reviewer Guidelines
• Submission Site
• Peer Review Toolkit
• Open Access Options
• Reasons to Publish
• Self-Archiving Policy
• About Clinical and Experimental Dermatology
• About British Association of Dermatologists
• Editorial Board
• Advertising & Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Discussion and conclusions, learning points, funding sources, data availability, ethics statement, cpd questions, instructions for answering questions.

• < Previous

How to critically appraise a systematic review: an aide for the reader and reviewer

Conflicts of interest H.W. founded the Cochrane Skin Group in 1987 and was coordinator editor until 2018. The other authors declare they have no conflicts of interest.

• Article contents
• Figures & tables
• Supplementary Data

John Frewen, Marianne de Brito, Anjali Pathak, Richard Barlow, Hywel C Williams, How to critically appraise a systematic review: an aide for the reader and reviewer, Clinical and Experimental Dermatology , Volume 48, Issue 8, August 2023, Pages 854–859, https://doi.org/10.1093/ced/llad141

• Permissions Icon Permissions

The number of published systematic reviews has soared rapidly in recent years. Sadly, the quality of most systematic reviews in dermatology is substandard. With the continued increase in exposure to systematic reviews, and their potential to influence clinical practice, we sought to describe a sequence of useful tips for the busy clinician reader to determine study quality and clinical utility. Important factors to consider when assessing systematic reviews include: determining the motivation to performing the study, establishing if the study protocol was prepublished, assessing quality of reporting using the PRISMA checklist, assessing study quality using the AMSTAR 2 critical appraisal checklist, assessing for evidence of spin, and summarizing the main strengths and limitations of the study to determine if it could change clinical practice. Having a set of heuristics to consider when reading systematic reviews serves to save time, enabling assessment of quality in a structured way, and come to a prompt conclusion of the merits of a review article in order to inform the care of dermatology patients.

A systematic review aims to systematically and transparently summarize the available data on a defined clinical question, via a rigorous search for studies, a critique of the quality of included studies and a qualitative and/or quantitative synthesis. 1 Systematic reviews are at the top of the pyramid in most evidence hierarchies for informing evidence-based healthcare as they are considered of greater validity and clinical applicability than those study types lower down, such as case series or individual trials. 2

A good systematic review should provide an unbiased overview of studies to inform clinical practice. Systematic reviews can reconcile apparently conflicting results, add precision to estimating smaller treatment effects, highlight the evidence’s limitations and biases and identify research gaps. Guidelines are available to assist systematic reviewers to transparently report why the review was done, the authors’ methods and findings via the PRISMA checklist. 3

The sharp rise in systematic review publications over time raises concern that the majority are unnecessary, misleading and/or conflicted. 4 A review of dermatology systematic reviews noted that 93% failed to report at least one PRISMA checklist item. 5 Another review of a random sample of 140/732 dermatology systematic reviews in 2017 found 90% were low quality. 6 Some improvements have occurred: reporting standards compliance has improved slightly (between 2013 and 2017), 5 and several leading dermatology journals including the British Journal of Dermatology have changed editorial policies, mandating authors to preregister review protocols.

Given the surge in poor-quality systematic review publications, we sought to describe a checklist of seven practical tips from the authors’ collective experience of writing and critically appraising systematic reviews, hoping that they will assist busy clinicians to critically appraise systematic reviews both as manuscript reviewers and as readers and research users.

Read the abstract to develop a sense of the subject.

What was the motivation for completing the review?

Has the review protocol been published and have changes been made to it.

Review the reporting quality .

Review the quality of the article and the depth of the review question.

Consider the authors’ interpretation and assess for spin .

Summarize and come to a position .

Read the abstract to develop a sense of the subject

From the abstract, use the PICO (population, intervention, comparator and outcome) framework to establish if the subject, intervention and outcomes are relevant to clinical practice. Is the review question clear and appropriate?

Inspect the authors’ conflicts of interest and funding sources. Self-disclosed financial conflicts are often insufficiently described or not declared at all. 7 If you suspect conflicts for authors with no stated conflicts, briefly searching the senior authors’ names on PubMed, or the Open Payments website (for US authors) may reveal hidden conflicts. 8 Is the motivation for the systematic review justified in the introduction? Can new insights be formed by combining studies? If the systematic review is an update, what new available data justifies this? Search for similar recent systematic reviews (which may have been omitted intentionally). Is it a redundant duplicate review that adds little new useful information? 9 Has the author recently published reviews on similar subjects? Salami publications refer to authors chopping up a topic into smaller pieces to obtain maximum publications. 10

Search PROSPERO for publication of the review protocol. 11 A prepublished review protocol in a publicly accessible site offers reassurance that the systematic review followed a clear plan with prespecified PICO elements. Put bluntly, it reduces authors’ opportunity for deception by selective analysis and highlighting of results that are more likely to get published. If a protocol is found, assess deviation from this protocol and justification, if present. Protocol registration allows improved PRISMA reporting. 12 A registered protocol with reporting of deviations allows the reader to judge whether any modifications are justified, for example adjusting for unexpected challenges during analysis. 10

Review the reporting quality

Look for supplementary material detailing the PRISMA checklist. Commonly under-reported PRISMA items include protocol and registration, risk of bias across studies, risk of bias in individual studies, the data collection process and review objectives. 5 Adequate reporting quality using PRISMA does not necessarily indicate the review is clinically useful; however, it allows the reader to assess the study’s utility (see Table 1 ). Additional assessments of review quality are described below.

The relationship between systematic review reporting quality and study quality a

Adapted with permission from Williams. 21

Review the quality of the article and the depth of the review question

Distinct from quality of reporting completeness, assessing the review's quality allows for assessment of the overall clinical meaningfulness of the results. Does the PICO make sense in respect to this? The AMSTAR 2 critical appraisal instrument is useful in determining quantitative systematic review quality. 13 This checklist marks the key aspects of a systematic review and computes an outcome of the review quality. 14 If meta-analysis was performed, did the authors justify and use appropriate methods for statistical combination of results? Were weighted techniques used to combine results and adjusted for heterogeneity, if present? If heterogeneity was present, were sources of this investigated? Did authors assess the potential impact of the individual study’s risk of bias (RoB) and perform analysis to investigate the impact of RoB on the summary estimates of affect? See Table 2 for an example of a completed AMSTAR 2 checklist on a recently published poor-quality systematic review. 15

An example of assessment of the quality of a systematic review (Drake et al. ) 15 using the AMSTAR 2 checklist an explanation of which can be found at https://amstar.ca/Amstar-2.php

N/A, not applicable; PICO, population, intervention, comparator and outcome. a Denotes AMSTAR 2 critical domain. The overall confidence in the results of the review is dependent on such critical domains. When one critical domain is not satisfied, the confidence is rated as ‘low’ and the review may not provide an accurate and comprehensive summary of the available studies that address the question of interest. When more than one critical domain are not satisfied, the confidence in the results of the review is rated as ‘critically low’ and the review should not be relied on to provide an accurate and comprehensive summary of the available studies.

Quality checklists for assessment of qualitative research include Consolidated Criteria for Reporting Qualitative research (COREQ), Standards for Reporting Qualitative Research (SRQR) and Critical Appraisal Skills Programme (CASP). 16 Such checklists aim to improve identification of high-quality qualitative research in journal articles, as well as acting as a guide for conducting research. 16

Consider the authors’ interpretation and assess for spin

Spin is a distorted interpretation of results. This manifests itself in studies as (i) misleading reporting, (ii) misleading interpretation, and (iii) inappropriate extrapolation. 14 Are the conclusion’s clinical practice recommendations not supported by the studies’ findings? Is the title misleading? Is there selective reporting? These are the three most severe forms of spin occurring in systematic reviews. 17

Summarize and come to a position

Summarize the reviews main positives and negatives and establish if there is sufficient quality to merit changing clinical practice, or are fatal flaws present that nullify the review’s clinical utility? Consider internal validity (are the results true?) and external validity (are the results applicable to my patient group?). When applying the systematic review results to a particular patient, it may help to consider these points: (i) how similar are the study participants to my patient?; (ii) do the outcomes make sense to me?; (iii) what was the magnitude of treatment benefit? – work out the number needed to treat; 18 (iv) what are the adverse events?; and (v) what are my patient's values and preferences? 19

Although systematic reviews have potential for summarizing evidence for dermatological interventions in a systematic and unbiased way, the rapid expansion of poorly reported and poor-quality reviews (Table 3 ) is regrettable. We do not claim our checklist items (Table 4 ) are superior to other checklists such as those suggested by CASP, 20 but they are based on the practical experience of critical appraisal of dermatology systematic reviews conducted by the authors.

The top seven ‘sins’ of dermatology systematic reviews a

Adapted with permission from Williams. 10

Checklist of questions, considerations and tips for critical appraisal of systematic reviews

NNT, number needed to treat.

Considering each question suggested in our checklist when faced with yet another systematic review draws a timely conclusion on its quality and application to clinical practice, when acting as a reviewer or reader. Although the checklist may sound exhaustive and time-consuming, we recommend cutting it short if there are major red flags early on, such as absence of a protocol or assessment of RoB. Given the growing number of systematic reviews, having an efficient and succinct aide for appraising articles saves the reader time and energy, while simplifying the decision regarding what merits a change in clinical practice. Our intention is not to criticize others’ well-intentioned efforts, but to improve standards of reliable evidence to inform patient care.

Systematic reviews of randomized controlled trials offer one of the best methods to summarize the evidence surrounding therapeutic interventions for skin conditions.

The number of systematic reviews in the dermatology literature is increasing rapidly.

The quality of dermatology systematic reviews is generally poor.

We describe a checklist for the busy clinician or reviewer to consider when faced with a systematic review.

Key factors to consider include: determining the review motivation, establishing if the study protocol was prepublished, assessing quality of reporting and study quality using PRISMA, and AMSTAR 2 critical appraisal checklists, and assessing for evidence of spin.

Summarizing the main qualities and limitations of a systematic review will help to determine if the review is robust enough to potentially change clinical practice for patient benefit.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

No new data generated.

Ethical approval: not applicable. Informed consent: not applicable.

Moher D , Liberati A , Tetzlaff J et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Ann Intern Med 2009 ; 151 : 264 – 9 .

Google Scholar

Murad MH , Asi N , Alsawas M , Alahdab F . New evidence pyramid . Evid Based Med 2016 ; 21 : 125 – 7 .

Page MJ , McKenzie JE , Bossuyt PM et al.  The PRISMA 2020 statement: an updated guideline for reporting systematic reviews . BMJ 2021 ; 372 : n71 .

Ioannidis JP . The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses . Milbank Q 2016 ; 94 : 485 – 514 .

Croitoru DO , Huang Y , Kurdina A et al.  Quality of reporting in systematic reviews published in dermatology journals . Br J Dermatol 2020 ; 182 : 1469 – 76 .

Smires S , Afach S , Mazaud C et al.  Quality and reporting completeness of systematic reviews and meta-analyses in dermatology . J Invest Dermatol 2021 ; 141 : 64 – 71 .

Baraldi JH , Picozzo SA , Arnold JC et al.  A cross-sectional examination of conflict-of-interest disclosures of physician-authors publishing in high-impact US medical journals . BMJ Open 2022 ; 12 : e057598 .

Centers for Medicare & Medicaid Services . Open Payments Search Tool. About. Available at : https://openpaymentsdata.cms.gov/about (last accessed 22 April 2023).

Guelimi R , Afach S , Régnaux JP et al.  Overlapping network meta-analyses on psoriasis systemic treatments, an overview: quantity does not make quality . Br J Dermatol 2022 ; 187 : 29 – 41 .

Williams HC . Are dermatology systematic reviews spinning out of control? Dermatology 2021 ; 237 : 493 – 5 .

National Institute for Health and Care Research . About Prospero. Available at : https://www.crd.york.ac.uk/prospero/#aboutpage (last accessed 22 April 2023).

Barbieri JS , Wehner MR . Systematic reviews in dermatology: opportunities for improvement . Br J Dermatol 2020 ; 182 : 1329 – 30 .

Shea BJ , Reeves BC , Wells G et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both . BMJ 2017 ; 358 : j4008 .

AMSTAR . AMSTAR checklist. Available at : https://amstar.ca/Amstar_Checklist.php (last accessed 22 April 2023).

Drake L , Reyes-Hadsall S , Martinez J et al.  Evaluation of the safety and effectiveness of nutritional supplements for treating hair loss: a systematic review . JAMA Dermatol 2023 ; 159 : 79 – 86 .

Stenfors T , Kajamaa A , Bennett D . How to … assess the quality of qualitative research . Clin Teach 2020 ; 17 : 596 – 9 .

Yavchitz A , Ravaud P , Altman DG et al.  A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity . J Clin Epidemiol 2016 ; 75 : 56 – 65 .

Manriquez JJ , Villouta MF , Williams HC . Evidence-based dermatology: number needed to treat and its relation to other risk measures . J Am Acad Dermatol 2007 ; 56 : 664 – 71 .

Williams HC . Applying trial evidence back to the patient . Arch Dermatol 2003 ; 139 : 1195 – 200 .

CASP . CASP checklists. Available at : https://casp-uk.net/casp-tools-checklists/  (last accessed 22 April 2023).

Williams HC . Cars, CONSORT 2010, and clinical practice . Trials 2010 ; 11 : 33 .

Learning objective

To demonstrate up-to-date knowledge on assessing systematic reviews.

Which of the following critical appraisal checklists is useful for assessment of items that should be reported in a systematic review?

Which one of the following statements is correct?

The number of published systematic reviews in the dermatology literature is falling.

The quality of published dermatology systematic reviews is generally very good.

Publishing details of the PRISMA checklist in a systematic review indicates that the study quality is high.

External validity refers to the applicability of results to your patient group.

Internal validity refers to the applicability of results to your patient group.

Spin in systematic reviews can be described by which one of the following measures?

Authors declaring all conflicts of interest.

Title suggesting beneficial effect not supported by findings.

Adequate reporting of study limitations.

Conclusion formulating recommendations for clinical practice supported by findings.

Reporting a departure from study protocol that may modify interpretation of results.

PICO stands for which of the following.

PubMed, inclusion, comparator, outcome.

Population, items, comparator, outcome.

Population, intervention, context, observations.

Protocol, intervention, certainty, outcome.

Population, intervention, comparator, outcome.

Publication of a systematic review study protocol can be found at which source?

Cochrane Library.

ClinicalTrials.gov.

This learning activity is freely available online at https://oupce.rievent.com/a/TWWDCK

Users are encouraged to

Read the article in print or online, paying particular attention to the learning points and any author conflict of interest disclosures.

Reflect on the article.

Register or login online at https://oupce.rievent.com/a/TWWDCK and answer the CPD questions.

Complete the required evaluation component of the activity.

Once the test is passed, you will receive a certificate and the learning activity can be added to your RCP CPD diary as a self-certified entry.

This activity will be available for CPD credit for 5 years following its publication date. At that time, it will be reviewed and potentially updated and extended for an additional period.

Author notes

Email alerts, citing articles via.

• Recommend to Your Librarian
• Advertising and Corporate Services
• Journals Career Network

Affiliations

• Online ISSN 1365-2230
• Copyright © 2024 British Association of Dermatologists
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

Critical Appraisal tools

Critical appraisal worksheets to help you appraise the reliability, importance and applicability of clinical evidence.

Critical appraisal is the systematic evaluation of clinical research papers in order to establish:

• Does this study address a  clearly focused question ?
• Did the study use valid methods to address this question?
• Are the valid results of this study important?
• Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you can save yourself the trouble of reading the rest of it.

This section contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples.

Critical Appraisal Worksheets

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostics  Critical Appraisal Sheet
• Prognosis  Critical Appraisal Sheet
• Randomised Controlled Trials  (RCT) Critical Appraisal Sheet
• Critical Appraisal of Qualitative Studies  Sheet
• IPD Review  Sheet

Chinese - translated by Chung-Han Yang and Shih-Chieh Shao

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostic Study  Critical Appraisal Sheet
• Prognostic Critical Appraisal Sheet
• RCT  Critical Appraisal Sheet
• IPD reviews Critical Appraisal Sheet
• Qualitative Studies Critical Appraisal Sheet 

German - translated by Johannes Pohl and Martin Sadilek

• Systematic Review  Critical Appraisal Sheet
• Diagnosis Critical Appraisal Sheet
• Prognosis Critical Appraisal Sheet
• Therapy / RCT Critical Appraisal Sheet

Lithuanian - translated by Tumas Beinortas

• Systematic review appraisal Lithuanian (PDF)
• Diagnostic accuracy appraisal Lithuanian  (PDF)
• Prognostic study appraisal Lithuanian  (PDF)
• RCT appraisal sheets Lithuanian  (PDF)

Portugese - translated by Enderson Miranda, Rachel Riera and Luis Eduardo Fontes

• Portuguese – Systematic Review Study Appraisal Worksheet
• Portuguese – Diagnostic Study Appraisal Worksheet
• Portuguese – Prognostic Study Appraisal Worksheet
• Portuguese – RCT Study Appraisal Worksheet
• Portuguese – Systematic Review Evaluation of Individual Participant Data Worksheet
• Portuguese – Qualitative Studies Evaluation Worksheet

Spanish - translated by Ana Cristina Castro

• Systematic Review  (PDF)
• Diagnosis  (PDF)
• Prognosis  Spanish Translation (PDF)
• Therapy / RCT  Spanish Translation (PDF)

Persian - translated by Ahmad Sofi Mahmudi

• Prognosis  (PDF)
• PICO  Critical Appraisal Sheet (PDF)
• PICO Critical Appraisal Sheet (MS-Word)
• Educational Prescription  Critical Appraisal Sheet (PDF)

Explanations & Examples

• Pre-test probability
• SpPin and SnNout
• Likelihood Ratios

• University of Texas Libraries
• UT Libraries

Systematic Reviews & Evidence Synthesis Methods

Critical appraisal.

• Types of Reviews
• Formulate Question
• Find Existing Reviews & Protocols
• Register a Protocol
• Searching Systematically
• Search Hedges and Filters
• Supplementary Searching
• Managing Results
• Deduplication
• Coding and Data Extraction
• Glossary of Terms
• Librarian Support
• Systematic Review & Evidence Synthesis Boot Camp

Some reviews require a critical appraisal for each study that makes it through the screening process. This involves a risk of bias assessment and/or a quality assessment. The goal of these reviews is not just to find all of the studies, but to determine their methodological rigor, and therefore, their credibility.

"Critical appraisal is the balanced assessment of a piece of research, looking for its strengths and weaknesses and them coming to a balanced judgement about its trustworthiness and its suitability for use in a particular context." 1

It's important to consider the impact that poorly designed studies could have on your findings and to rule out inaccurate or biased work.

Selection of a valid critical appraisal tool, testing the tool with several of the selected studies, and involving two or more reviewers in the appraisal are good practices to follow.

1. Purssell E, McCrae N. How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. 1st ed. Springer ;  2020.

Evaluation Tools

• The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) was developed to address the issue of variability in the quality of practice guidelines.
• Centre for Evidence-Based Medicine (CEBM). Critical Appraisal Tools "contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples."
• Critical Appraisal Skills Programme (CASP) Checklists Critical Appraisal checklists for many different study types
• Critical Review Form for Qualitative Studies Version 2, developed out of McMaster University
• Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS) Downes MJ, Brennan ML, Williams HC, et al. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016;6:e011458. doi:10.1136/bmjopen-2016-011458
• Downs & Black Checklist for Assessing Studies Downs, S. H., & Black, N. (1998). The Feasibility of Creating a Checklist for the Assessment of the Methodological Quality Both of Randomised and Non-Randomised Studies of Health Care Interventions. Journal of Epidemiology and Community Health (1979-), 52(6), 377–384.
• GRADE The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group "has developed a common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations."
• Grade Handbook Full handbook on the GRADE method for grading quality of evidence.
• MAGIC (Making GRADE the Irresistible choice) Clear succinct guidance in how to use GRADE
• Joanna Briggs Institute. Critical Appraisal Tools "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers." Includes checklists for 13 types of articles.
• Latitudes Network This is a searchable library of validity assessment tools for use in evidence syntheses. This website also provides access to training on the process of validity assessment.
• Mixed Methods Appraisal Tool A tool that can be used to appraise a mix of studies that are included in a systematic review - qualitative research, RCTs, non-randomized studies, quantitative studies, mixed methods studies.
• RoB 2 Tool Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, Welch V (editors). Cochrane Methods. Cochrane Database of Systematic Reviews 2016, Issue 10 (Suppl 1). dx.doi.org/10.1002/14651858.CD201601.
• ROBINS-I Risk of Bias for non-randomized (observational) studies or cohorts of interventions Sterne J A, Hernán M A, Reeves B C, Savović J, Berkman N D, Viswanathan M et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions BMJ 2016; 355 :i4919 doi:10.1136/bmj.i4919
• Scottish Intercollegiate Guidelines Network. Critical Appraisal Notes and Checklists "Methodological assessment of studies selected as potential sources of evidence is based on a number of criteria that focus on those aspects of the study design that research has shown to have a significant effect on the risk of bias in the results reported and conclusions drawn. These criteria differ between study types, and a range of checklists is used to bring a degree of consistency to the assessment process."
• The TREND Statement (CDC) Des Jarlais DC, Lyles C, Crepaz N, and the TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. Am J Public Health. 2004;94:361-366.
• Assembling the Pieces of a Systematic Reviews, Chapter 8: Evaluating: Study Selection and Critical Appraisal.
• How to Perform a Systematic Literature Review, Chapter: Critical Appraisal: Assessing the Quality of Studies.

Other library guides

• Duke University Medical Center Library. Systematic Reviews: Assess for Quality and Bias
• UNC Health Sciences Library. Systematic Reviews: Assess Quality of Included Studies
• Last Updated: Nov 19, 2024 11:14 AM
• URL: https://guides.lib.utexas.edu/systematicreviews

Critical Appraisal of a Systematic Review: A Concise Review

Affiliations.

• 1 Division of Pulmonary and Critical Care Medicine, Department of Medicine, Medical College of Wisconsin, Milwaukee, WI.
• 2 Department of Anesthesiology, University Hospital RWTH Aachen University, Aachen, Germany.
• 3 Department of Intensive Care Medicine, University Hospital RWTH Aachen University, Aachen, Germany.
• 4 Department of Anesthesiology, Faculty of Medicine, University of Malaya, Kuala Lumpur, Malaysia.
• 5 Clinical Evaluation Research Unit, Department of Critical Care Medicine, Queen's University, KGH Research Institute, Kingston Health Sciences Centre, Kingston, ON, Canada.
• 6 Department of Anesthesiology and Intensive Care Medicine, University Hospital Wuerzburg, Wuerzburg, Germany.
• PMID: 35853198
• DOI: 10.1097/CCM.0000000000005602

Objectives: Concise definitive review of how to read and critically appraise a systematic review.

Data sources: None.

Study selection: Current literature describing the conduct, reporting, and appraisal of systematic reviews and meta-analyses.

Data extraction: Best practices for conducting, reporting, and appraising systematic review were summarized.

Data synthesis: A systematic review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant original research, and to collect and analyze data from the studies that are included in the review. Critical appraisal methods address both the credibility (quality of conduct) and rate the confidence in the quality of summarized evidence from a systematic review. The A Measurement Tool to Assess Systematic Reviews-2 tool is a widely used practical tool to appraise the conduct of a systematic review. Confidence in estimates of effect is determined by assessing for risk of bias, inconsistency of results, imprecision, indirectness of evidence, and publication bias.

Conclusions: Systematic reviews are transparent and reproducible summaries of research and conclusions drawn from them are only as credible and reliable as their development process and the studies which form the systematic review. Applying evidence from a systematic review to patient care considers whether the results can be directly applied, whether all important outcomes have been considered, and if the benefits are worth potential harms and costs.

Copyright © 2022 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.

Publication types

• Publications
• Systematic Reviews as Topic*

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

Critical Appraisal Toolkit (CAT) for assessing multiple types of evidence

• Author information
• Article notes
• Copyright and License information

Correspondence: [email protected]

Contributor: Jennifer Kruse, Public Health Agency of Canada – Conceptualization and project administration

Series information

Scientific writing

Collection date 2017 Sep 7.

Healthcare professionals are often expected to critically appraise research evidence in order to make recommendations for practice and policy development. Here we describe the Critical Appraisal Toolkit (CAT) currently used by the Public Health Agency of Canada. The CAT consists of: algorithms to identify the type of study design, three separate tools (for appraisal of analytic studies, descriptive studies and literature reviews), additional tools to support the appraisal process, and guidance for summarizing evidence and drawing conclusions about a body of evidence. Although the toolkit was created to assist in the development of national guidelines related to infection prevention and control, clinicians, policy makers and students can use it to guide appraisal of any health-related quantitative research. Participants in a pilot test completed a total of 101 critical appraisals and found that the CAT was user-friendly and helpful in the process of critical appraisal. Feedback from participants of the pilot test of the CAT informed further revisions prior to its release. The CAT adds to the arsenal of available tools and can be especially useful when the best available evidence comes from non-clinical trials and/or studies with weak designs, where other tools may not be easily applied.

Introduction

Healthcare professionals, researchers and policy makers are often involved in the development of public health policies or guidelines. The most valuable guidelines provide a basis for evidence-based practice with recommendations informed by current, high quality, peer-reviewed scientific evidence. To develop such guidelines, the available evidence needs to be critically appraised so that recommendations are based on the "best" evidence. The ability to critically appraise research is, therefore, an essential skill for health professionals serving on policy or guideline development working groups.

Our experience with working groups developing infection prevention and control guidelines was that the review of relevant evidence went smoothly while the critical appraisal of the evidence posed multiple challenges. Three main issues were identified. First, although working group members had strong expertise in infection prevention and control or other areas relevant to the guideline topic, they had varying levels of expertise in research methods and critical appraisal. Second, the critical appraisal tools in use at that time focused largely on analytic studies (such as clinical trials), and lacked definitions of key terms and explanations of the criteria used in the studies. As a result, the use of these tools by working group members did not result in a consistent way of appraising analytic studies nor did the tools provide a means of assessing descriptive studies and literature reviews. Third, working group members wanted guidance on how to progress from assessing individual studies to summarizing and assessing a body of evidence.

To address these issues, a review of existing critical appraisal tools was conducted. We found that the majority of existing tools were design-specific, with considerable variability in intent, criteria appraised and construction of the tools. A systematic review reported that fewer than half of existing tools had guidelines for use of the tool and interpretation of the items ( 1 ). The well-known Grading of Recommendations Assessment, Development and Evaluation (GRADE) rating-of-evidence system and the Cochrane tools for assessing risk of bias were considered for use ( 2 ), ( 3 ). At that time, the guidelines for using these tools were limited, and the tools were focused primarily on randomized controlled trials (RCTs) and non-randomized controlled trials. For feasibility and ethical reasons, clinical trials are rarely available for many common infection prevention and control issues ( 4 ), ( 5 ). For example, there are no intervention studies assessing which practice restrictions, if any, should be placed on healthcare workers who are infected with a blood-borne pathogen. Working group members were concerned that if they used GRADE, all evidence would be rated as very low or as low quality or certainty, and recommendations based on this evidence may be interpreted as unconvincing, even if they were based on the best or only available evidence.

The team decided to develop its own critical appraisal toolkit. So a small working group was convened, led by an epidemiologist with expertise in research, methodology and critical appraisal, with the goal of developing tools to critically appraise studies informing infection prevention and control recommendations. This article provides an overview of the Critical Appraisal Toolkit (CAT). The full document, entitled Infection Prevention and Control Guidelines Critical Appraisal Tool Kit is available online ( 6 ).

Following a review of existing critical appraisal tools, studies informing infection prevention and control guidelines that were in development were reviewed to identify the types of studies that would need to be appraised using the CAT. A preliminary draft of the CAT was used by various guideline development working groups and iterative revisions were made over a two year period. A pilot test of the CAT was then conducted which led to the final version ( 6 ).

The toolkit is set up to guide reviewers through three major phases in the critical appraisal of a body of evidence: appraisal of individual studies; summarizing the results of the appraisals; and appraisal of the body of evidence.

Tools for critically appraising individual studies

The first step in the critical appraisal of an individual study is to identify the study design; this can be surprisingly problematic, since many published research studies are complex. An algorithm was developed to help identify whether a study was an analytic study, a descriptive study or a literature review (see text box for definitions). It is critical to establish the design of the study first, as the criteria for assessment differs depending on the type of study.

Definitions of the types of studies that can be analyzed with the Critical Appraisal Toolkit*

Analytic study: A study designed to identify or measure effects of specific exposures, interventions or risk factors. This design employs the use of an appropriate comparison group to test epidemiologic hypotheses, thus attempting to identify associations or causal relationships.

Descriptive study: A study that describes characteristics of a condition in relation to particular factors or exposure of interest. This design often provides the first important clues about possible determinants of disease and is useful for the formulation of hypotheses that can be subsequently tested using an analytic design.

Literature review: A study that analyzes critical points of a published body of knowledge. This is done through summary, classification and comparison of prior studies. With the exception of meta-analyses, which statistically re-analyze pooled data from several studies, these studies are secondary sources and do not report any new or experimental work.

* Public Health Agency of Canada. Infection Prevention and Control Guidelines Critical Appraisal Tool Kit ( 6 )

Separate algorithms were developed for analytic studies, descriptive studies and literature reviews to help reviewers identify specific designs within those categories. The algorithm below, for example, helps reviewers determine which study design was used within the analytic study category ( Figure 1 ). It is based on key decision points such as number of groups or allocation to group. The legends for the algorithms and supportive tools such as the glossary provide additional detail to further differentiate study designs, such as whether a cohort study was retrospective or prospective.

Figure 1. Algorithm for identifying the type of analytic study.

Abbreviations: CBA, controlled before-after; ITS, interrupted time series; NRCT, non-randomized controlled trial; RCT, randomized controlled trial; UCBA, uncontrolled before-after

Separate critical appraisal tools were developed for analytic studies, for descriptive studies and for literature reviews, with relevant criteria in each tool. For example, a summary of the items covered in the analytic study critical appraisal tool is shown in Table 1 . This tool is used to appraise trials, observational studies and laboratory-based experiments. A supportive tool for assessing statistical analysis was also provided that describes common statistical tests used in epidemiologic studies.

Table 1. Aspects appraised in analytic study critical appraisal tool.

The descriptive study critical appraisal tool assesses different aspects of sampling, data collection, statistical analysis, and ethical conduct. It is used to appraise cross-sectional studies, outbreak investigations, case series and case reports.

The literature review critical appraisal tool assesses the methodology, results and applicability of narrative reviews, systematic reviews and meta-analyses.

After appraisal of individual items in each type of study, each critical appraisal tool also contains instructions for drawing a conclusion about the overall quality of the evidence from a study, based on the per-item appraisal. Quality is rated as high, medium or low. While a RCT is a strong study design and a survey is a weak design, it is possible to have a poor quality RCT or a high quality survey. As a result, the quality of evidence from a study is distinguished from the strength of a study design when assessing the quality of the overall body of evidence. A definition of some terms used to evaluate evidence in the CAT is shown in Table 2 .

Table 2. Definition of terms used to evaluate evidence.

* Considered strong design if there are at least two control groups and two intervention groups. Considered moderate design if there is only one control and one intervention group

Tools for summarizing the evidence

The second phase in the critical appraisal process involves summarizing the results of the critical appraisal of individual studies. Reviewers are instructed to complete a template evidence summary table, with key details about each study and its ratings. Studies are listed in descending order of strength in the table. The table simplifies looking across all studies that make up the body of evidence informing a recommendation and allows for easy comparison of participants, sample size, methods, interventions, magnitude and consistency of results, outcome measures and individual study quality as determined by the critical appraisal. These evidence summary tables are reviewed by the working group to determine the rating for the quality of the overall body of evidence and to facilitate development of recommendations based on evidence.

Rating the quality of the overall body of evidence

The third phase in the critical appraisal process is rating the quality of the overall body of evidence. The overall rating depends on the five items summarized in Table 2 : strength of study designs, quality of studies, number of studies, consistency of results and directness of the evidence. The various combinations of these factors lead to an overall rating of the strength of the body of evidence as strong, moderate or weak as summarized in Table 3 .

Table 3. Criteria for rating evidence on which recommendations are based.

A unique aspect of this toolkit is that recommendations are not graded but are formulated based on the graded body of evidence. Actions are either recommended or not recommended; it is the strength of the available evidence that varies, not the strength of the recommendation. The toolkit does highlight, however, the need to re-evaluate new evidence as it becomes available especially when recommendations are based on weak evidence.

Pilot test of the CAT

Of 34 individuals who indicated an interest in completing the pilot test, 17 completed it. Multiple peer-reviewed studies were selected representing analytic studies, descriptive studies and literature reviews. The same studies were assigned to participants with similar content expertise. Each participant was asked to appraise three analytic studies, two descriptive studies and one literature review, using the appropriate critical appraisal tool as identified by the participant. For each study appraised, one critical appraisal tool and the associated tool-specific feedback form were completed. Each participant also completed a single general feedback form. A total of 101 of 102 critical appraisals were conducted and returned, with 81 tool-specific feedback forms and 14 general feedback forms returned.

The majority of participants (>85%) found the flow of each tool was logical and the length acceptable but noted they still had difficulty identifying the study designs ( Table 4 ).

Table 4. Pilot test feedback on user friendliness.

* Number of tool-specific forms returned for total number of critical appraisals conducted

The vast majority of the feedback forms (86–93%) indicated that the different tools facilitated the critical appraisal process. In the assessment of consistency, however, only four of ten analytic studies appraised (40%), had complete agreement on the rating of overall study quality by participants, the other six studies had differences noted as mismatches. Four of the six studies with mismatches were observational studies. The differences were minor. None of the mismatches included a study that was rated as both high and low quality by different participants. Based on the comments provided by participants, most mismatches could likely have been resolved through discussion with peers. Mismatched ratings were not an issue for the descriptive studies and literature reviews. In summary, the pilot test provided useful feedback on different aspects of the toolkit. Revisions were made to address the issues identified from the pilot test and thus strengthen the CAT.

The Infection Prevention and Control Guidelines Critical Appraisal Tool Kit was developed in response to the needs of infection control professionals reviewing literature that generally did not include clinical trial evidence. The toolkit was designed to meet the identified needs for training in critical appraisal with extensive instructions and dictionaries, and tools applicable to all three types of studies (analytic studies, descriptive studies and literature reviews). The toolkit provided a method to progress from assessing individual studies to summarizing and assessing the strength of a body of evidence and assigning a grade. Recommendations are then developed based on the graded body of evidence. This grading system has been used by the Public Health Agency of Canada in the development of recent infection prevention and control guidelines ( 5 ), ( 7 ). The toolkit has also been used for conducting critical appraisal for other purposes, such as addressing a practice problem and serving as an educational tool ( 8 ), ( 9 ).

The CAT has a number of strengths. It is applicable to a wide variety of study designs. The criteria that are assessed allow for a comprehensive appraisal of individual studies and facilitates critical appraisal of a body of evidence. The dictionaries provide reviewers with a common language and criteria for discussion and decision making.

The CAT also has a number of limitations. The tools do not address all study designs (e.g., modelling studies) and the toolkit provides limited information on types of bias. Like the majority of critical appraisal tools ( 10 ), ( 11 ), these tools have not been tested for validity and reliability. Nonetheless, the criteria assessed are those indicated as important in textbooks and in the literature ( 12 ), ( 13 ). The grading scale used in this toolkit does not allow for comparison of evidence grading across organizations or internationally, but most reviewers do not need such comparability. It is more important that strong evidence be rated higher than weak evidence, and that reviewers provide rationales for their conclusions; the toolkit enables them to do so.

Overall, the pilot test reinforced that the CAT can help with critical appraisal training and can increase comfort levels for those with limited experience. Further evaluation of the toolkit could assess the effectiveness of revisions made and test its validity and reliability.

A frequent question regarding this toolkit is how it differs from GRADE as both distinguish stronger evidence from weaker evidence and use similar concepts and terminology. The main differences between GRADE and the CAT are presented in Table 5 . Key differences include the focus of the CAT on rating the quality of individual studies, and the detailed instructions and supporting tools that assist those with limited experience in critical appraisal. When clinical trials and well controlled intervention studies are or become available, GRADE and related tools from Cochrane would be more appropriate ( 2 ), ( 3 ). When descriptive studies are all that is available, the CAT is very useful.

Table 5. Comparison of features of the Critical Appraisal Toolkit (CAT) and GRADE.

Abbreviation: GRADE, Grading of Recommendations Assessment, Development and Evaluation

The Infection Prevention and Control Guidelines Critical Appraisal Tool Kit was developed in response to needs for training in critical appraisal, assessing evidence from a wide variety of research designs, and a method for going from assessing individual studies to characterizing the strength of a body of evidence. Clinician researchers, policy makers and students can use these tools for critical appraisal of studies whether they are trying to develop policies, find a potential solution to a practice problem or critique an article for a journal club. The toolkit adds to the arsenal of critical appraisal tools currently available and is especially useful in assessing evidence from a wide variety of research designs.

Authors’ Statement

DM – Conceptualization, methodology, investigation, data collection and curation and writing – original draft, review and editing

TO – Conceptualization, methodology, investigation, data collection and curation and writing – original draft, review and editing

KD – Conceptualization, review and editing, supervision and project administration

Acknowledgements

We thank the Infection Prevention and Control Expert Working Group of the Public Health Agency of Canada for feedback on the development of the toolkit, Lisa Marie Wasmund for data entry of the pilot test results, Katherine Defalco for review of data and cross-editing of content and technical terminology for the French version of the toolkit, Laurie O’Neil for review and feedback on early versions of the toolkit, Frédéric Bergeron for technical support with the algorithms in the toolkit and the Centre for Communicable Diseases and Infection Control of the Public Health Agency of Canada for review, feedback and ongoing use of the toolkit. We thank Dr. Patricia Huston, Canada Communicable Disease Report Editor-in-Chief, for a thorough review and constructive feedback on the draft manuscript.

Conflict of interest: None.

Funding: This work was supported by the Public Health Agency of Canada.

• 1. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 2004. Sep;4:22. 10.1186/1471-2288-4-22 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 2. GRADE Working Group. Criteria for applying or using GRADE. www.gradeworkinggroup.org [Accessed July 25, 2017].
• 3. Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration, 2011. http://handbook.cochrane.org
• 4. Khan AR, Khan S, Zimmerman V, Baddour LM, Tleyjeh IM. Quality and strength of evidence of the Infectious Diseases Society of America clinical practice guidelines. Clin Infect Dis 2010. Nov;51(10):1147–56. 10.1086/656735 [ DOI ] [ PubMed ] [ Google Scholar ]
• 5. Public Health Agency of Canada. Routine practices and additional precautions for preventing the transmission of infection in healthcare settings. http://www.phac-aspc.gc.ca/nois-sinp/guide/summary-sommaire/tihs-tims-eng.php [Accessed December 4, 2015].
• 6. Public Health Agency of Canada. Infection Prevention and Control Guidelines Critical Appraisal Tool Kit. http://publications.gc.ca/collections/collection_2014/aspc-phac/HP40-119-2014-eng.pdf [Accessed December 4, 2015].
• 7. Public Health Agency of Canada. Hand hygiene practices in healthcare settings. https://www.canada.ca/en/public-health/services/infectious-diseases/nosocomial-occupational-infections/hand-hygiene-practices-healthcare-settings.html [Accessed December 4, 2015].
• 8. Ha S, Paquette D, Tarasuk J, Dodds J, Gale-Rowe M, Brooks JI et al. A systematic review of HIV testing among Canadian populations. Can J Public Health 2014. Jan;105(1):e53–62. 10.17269/cjph.105.4128 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 9. Stevens LK, Ricketts ED, Bruneau JE. Critical appraisal through a new lens. Nurs Leadersh (Tor Ont) 2014. Jun;27(2):10–3. 10.12927/cjnl.2014.23843 [ DOI ] [ PubMed ] [ Google Scholar ]
• 10. Lohr KN. Rating the strength of scientific evidence: relevance for quality improvement programs. Int J Qual Health Care 2004. Feb;16(1):9–18. 10.1093/intqhc/mzh005 [ DOI ] [ PubMed ] [ Google Scholar ]
• 11. Crowe M, Sheppard L. A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. J Clin Epidemiol 2011. Jan;64(1):79–89. 10.1016/j.jclinepi.2010.02.008 [ DOI ] [ PubMed ] [ Google Scholar ]
• 12. Young JM, Solomon MJ. How to critically appraise an article. Nat Clin Pract Gastroenterol Hepatol 2009. Feb;6(2):82–91. 10.1038/ncpgasthep1331 [ DOI ] [ PubMed ] [ Google Scholar ]
• 13. Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice. 9 th ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2008. Chapter XX, Literature reviews: Finding and critiquing the evidence p. 94-125. [ Google Scholar ]
• View on publisher site
• PDF (348.4 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

• Mayo Clinic Libraries
• Evidence Synthesis Guide
• Risk of Bias by Study Design

Evidence Synthesis Guide : Risk of Bias by Study Design

• Review Types & Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Minimize Bias
• GRADE & GRADE-CERQual
• Data Extraction Tools
• Synthesis & Meta-Analysis
• Publishing your Review
• Mayo Clinic Libraries Evidence Synthesis Service

Risk of Bias of Individual Studies

““Assessment of risk of bias is a key step that informs many other steps and decisions made in conducting systematic reviews. It plays an important role in the final assessment of the strength of the evidence.” 1  

Risk of Bias by Study Design (featured tools)

• Systematic Reviews
• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions. more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review. more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial, rather than the entire trial. more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report. more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units to comparison groups. more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. The AXIS tool focuses mainly on the presented study methods and results. more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess the robustness of the evidence of uncontrolled case series studies.
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool is designed to assess the quality of primary diagnostic accuracy studies it consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard. more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that essential elements of diagnostic accuracy study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible. The Standards for the Reporting of Diagnostic Accuracy Studies was developed to help improve completeness and transparency in reporting of diagnostic accuracy studies. more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB Implementation of SYRCLE’s RoB tool will facilitate and improve critical appraisal of evidence from animal studies. This may enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies. more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 The ARRIVE 2.0 guidelines are a checklist of information to include in a manuscript to ensure that publications on in vivo animal studies contain enough information to add to the knowledge base. more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides an approach to critically appraising papers based on the results of laboratory animal experiments, and discusses various bias domains in the literature that critical appraisal can identify.
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.    Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., ... & Kane, R. L. (2018). Recommendations for assessing the risk of bias in systematic reviews of health-care interventions .  Journal of clinical epidemiology ,  97 , 26-34.

2.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

5..     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

6.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

7.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

8..    Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

9.    Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

10.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

11.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

12.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

13.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

14.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

15.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

16.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

17.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE & GRADE-CERQual >>
• Last Updated: Nov 11, 2024 10:27 AM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess