how to critically evaluate a qualitative research paper

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Critically appraising...

Critically appraising qualitative research

• Related content
• Peer review
• Ayelet Kuper , assistant professor 1 ,
• Lorelei Lingard , associate professor 2 ,
• Wendy Levinson , Sir John and Lady Eaton professor and chair 3
• 1 Department of Medicine, Sunnybrook Health Sciences Centre, and Wilson Centre for Research in Education, University of Toronto, 2075 Bayview Avenue, Room HG 08, Toronto, ON, Canada M4N 3M5
• 2 Department of Paediatrics and Wilson Centre for Research in Education, University of Toronto and SickKids Learning Institute; BMO Financial Group Professor in Health Professions Education Research, University Health Network, 200 Elizabeth Street, Eaton South 1-565, Toronto
• 3 Department of Medicine, Sunnybrook Health Sciences Centre
• Correspondence to: A Kuper ayelet94{at}post.harvard.edu

Six key questions will help readers to assess qualitative research

Summary points

Appraising qualitative research is different from appraising quantitative research

Qualitative research papers should show appropriate sampling, data collection, and data analysis

Transferability of qualitative research depends on context and may be enhanced by using theory

Ethics in qualitative research goes beyond review boards’ requirements to involve complex issues of confidentiality, reflexivity, and power

Over the past decade, readers of medical journals have gained skills in critically appraising studies to determine whether the results can be trusted and applied to their own practice settings. Criteria have been designed to assess studies that use quantitative methods, and these are now in common use.

In this article we offer guidance for readers on how to assess a study that uses qualitative research methods by providing six key questions to ask when reading qualitative research (box 1). However, the thorough assessment of qualitative research is an interpretive act and requires informed reflective thought rather than the simple application of a scoring system.

Box 1 Key questions to ask when reading qualitative research studies

Was the sample used in the study appropriate to its research question.

Were the data collected appropriately?

Were the data analysed appropriately?

Can I transfer the results of this study to my own setting?

Does the study adequately address potential ethical issues, including reflexivity?

Overall: is what the researchers did clear?

One of the critical decisions in a qualitative study is whom or what to include in the sample—whom to interview, whom to observe, what texts to analyse. An understanding that qualitative research is based in experience and in the construction of meaning, combined with the specific research question, should guide the sampling process. For example, a study of the experience of survivors of domestic violence that examined their reasons for not seeking help from healthcare providers might focus on interviewing a …

Log in using your username and password

BMA Member Log In

If you have a subscription to The BMJ, log in:

• Need to activate
• Log in via institution
• Log in via OpenAthens

Log in through your institution

Subscribe from £184 *.

Subscribe and get access to all BMJ articles, and much more.

* For online subscription

Access this article for 1 day for: £50 / $60/ €56 ( excludes VAT )

You can download a PDF version for your personal record.

Buy this article

• Teesside University Student & Library Services
• Learning Hub Group

Critical Appraisal for Health Students

• Critical Appraisal of a qualitative paper
• Critical Appraisal: Help
• Critical Appraisal of a quantitative paper
• Useful resources

Appraisal of a Qualitative paper : Top tips

• Introduction

Critical appraisal of a qualitative paper

This guide aimed at health students, provides basic level support for appraising qualitative research papers. It's designed for students who have already attended lectures on critical appraisal. One framework  for appraising qualitative research (based on 4 aspects of trustworthiness) is  provided and there is an opportunity to practise the technique on a sample article.

Support Materials

• Framework for reading qualitative papers
• Critical appraisal of a qualitative paper PowerPoint

To practise following this framework for critically appraising a qualitative article, please look at the following article:

Schellekens, M.P.J.  et al  (2016) 'A qualitative study on mindfulness-based stress reduction for breast cancer patients: how women experience participating with fellow patients',  Support Care Cancer , 24(4), pp. 1813-1820.

Critical appraisal of a qualitative paper: practical example.

• Credibility
• Transferability
• Dependability
• Confirmability

How to use this practical example 

Using the framework, you can have a go at appraising a qualitative paper - we are going to look at the following article: 

Step 1.  take a quick look at the article, step 2.  click on the credibility tab above - there are questions to help you appraise the trustworthiness of the article, read the questions and look for the answers in the article. , step 3.   click on each question and our answers will appear., step 4.    repeat with the other aspects of trustworthiness: transferability, dependability and confirmability ., questioning the credibility:, who is the researcher what has been their experience how well do they know this research area, was the best method chosen what method did they use was there any justification was the method scrutinised by peers is it a recognisable method was there triangulation ( more than one method used), how was the data collected was data collected from the participants at more than one time point how long were the interviews were questions asked to the participants in different ways, is the research reporting what the participants actually said were the participants shown transcripts / notes of the interviews / observations to ‘check’ for accuracy are direct quotes used from a variety of participants, how would you rate the overall credibility, questioning the transferability, was a meaningful sample obtained how many people were included is the sample diverse how were they selected, are the demographics given, does the research cover diverse viewpoints do the results include negative cases was data saturation reached, what is the overall transferability can the research be transferred to other settings , questioning the dependability :, how transparent is the audit trail can you follow the research steps are the decisions made transparent is the whole process explained in enough detail did the researcher keep a field diary is there a clear limitations section, was there peer scrutiny of the researchwas the research plan shown to peers / colleagues for approval and/or feedback did two or more researchers independently judge data, how would you rate the overall dependability would the results be similar if the study was repeated how consistent are the data and findings, questioning the confirmability :, is the process of analysis described in detail is a method of analysis named or described is there sufficient detail, have any checks taken place was there cross-checking of themes was there a team of researchers, has the researcher reflected on possible bias is there a reflexive diary, giving a detailed log of thoughts, ideas and assumptions, how do you rate the overall confirmability has the researcher attempted to limit bias, questioning the overall trustworthiness :, overall how trustworthy is the research, further information.

See Useful resources  for links, books and LibGuides to help with Critical appraisal.

• << Previous: Critical Appraisal: Help
• Next: Critical Appraisal of a quantitative paper >>
• Last Updated: Jul 23, 2024 3:37 PM
• URL: https://libguides.tees.ac.uk/critical_appraisal

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals

You are here

• Volume 22, Issue 1
• How to appraise qualitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Calvin Moorley 1 ,
• Xabi Cathala 2
• 1 Nursing Research and Diversity in Care, School of Health and Social Care , London South Bank University , London , UK
• 2 Institute of Vocational Learning , School of Health and Social Care, London South Bank University , London , UK
• Correspondence to Dr Calvin Moorley, Nursing Research and Diversity in Care, School of Health and Social Care, London South Bank University, London SE1 0AA, UK; Moorleyc{at}lsbu.ac.uk

https://doi.org/10.1136/ebnurs-2018-103044

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

In order to make a decision about implementing evidence into practice, nurses need to be able to critically appraise research. Nurses also have a professional responsibility to maintain up-to-date practice. 1 This paper provides a guide on how to critically appraise a qualitative research paper.

What is qualitative research?

• View inline

Useful terms

Some of the qualitative approaches used in nursing research include grounded theory, phenomenology, ethnography, case study (can lend itself to mixed methods) and narrative analysis. The data collection methods used in qualitative research include in depth interviews, focus groups, observations and stories in the form of diaries or other documents. 3

Authenticity

Title, keywords, authors and abstract.

In a previous paper, we discussed how the title, keywords, authors’ positions and affiliations and abstract can influence the authenticity and readability of quantitative research papers, 4 the same applies to qualitative research. However, other areas such as the purpose of the study and the research question, theoretical and conceptual frameworks, sampling and methodology also need consideration when appraising a qualitative paper.

Purpose and question

The topic under investigation in the study should be guided by a clear research question or a statement of the problem or purpose. An example of a statement can be seen in table 2 . Unlike most quantitative studies, qualitative research does not seek to test a hypothesis. The research statement should be specific to the problem and should be reflected in the design. This will inform the reader of what will be studied and justify the purpose of the study. 5

Example of research question and problem statement

An appropriate literature review should have been conducted and summarised in the paper. It should be linked to the subject, using peer-reviewed primary research which is up to date. We suggest papers with a age limit of 5–8 years excluding original work. The literature review should give the reader a balanced view on what has been written on the subject. It is worth noting that for some qualitative approaches some literature reviews are conducted after the data collection to minimise bias, for example, in grounded theory studies. In phenomenological studies, the review sometimes occurs after the data analysis. If this is the case, the author(s) should make this clear.

Theoretical and conceptual frameworks

Most authors use the terms theoretical and conceptual frameworks interchangeably. Usually, a theoretical framework is used when research is underpinned by one theory that aims to help predict, explain and understand the topic investigated. A theoretical framework is the blueprint that can hold or scaffold a study’s theory. Conceptual frameworks are based on concepts from various theories and findings which help to guide the research. 6 It is the researcher’s understanding of how different variables are connected in the study, for example, the literature review and research question. Theoretical and conceptual frameworks connect the researcher to existing knowledge and these are used in a study to help to explain and understand what is being investigated. A framework is the design or map for a study. When you are appraising a qualitative paper, you should be able to see how the framework helped with (1) providing a rationale and (2) the development of research questions or statements. 7 You should be able to identify how the framework, research question, purpose and literature review all complement each other.

There remains an ongoing debate in relation to what an appropriate sample size should be for a qualitative study. We hold the view that qualitative research does not seek to power and a sample size can be as small as one (eg, a single case study) or any number above one (a grounded theory study) providing that it is appropriate and answers the research problem. Shorten and Moorley 8 explain that three main types of sampling exist in qualitative research: (1) convenience (2) judgement or (3) theoretical. In the paper , the sample size should be stated and a rationale for how it was decided should be clear.

Methodology

Qualitative research encompasses a variety of methods and designs. Based on the chosen method or design, the findings may be reported in a variety of different formats. Table 3 provides the main qualitative approaches used in nursing with a short description.

Different qualitative approaches

The authors should make it clear why they are using a qualitative methodology and the chosen theoretical approach or framework. The paper should provide details of participant inclusion and exclusion criteria as well as recruitment sites where the sample was drawn from, for example, urban, rural, hospital inpatient or community. Methods of data collection should be identified and be appropriate for the research statement/question.

Data collection

Overall there should be a clear trail of data collection. The paper should explain when and how the study was advertised, participants were recruited and consented. it should also state when and where the data collection took place. Data collection methods include interviews, this can be structured or unstructured and in depth one to one or group. 9 Group interviews are often referred to as focus group interviews these are often voice recorded and transcribed verbatim. It should be clear if these were conducted face to face, telephone or any other type of media used. Table 3 includes some data collection methods. Other collection methods not included in table 3 examples are observation, diaries, video recording, photographs, documents or objects (artefacts). The schedule of questions for interview or the protocol for non-interview data collection should be provided, available or discussed in the paper. Some authors may use the term ‘recruitment ended once data saturation was reached’. This simply mean that the researchers were not gaining any new information at subsequent interviews, so they stopped data collection.

The data collection section should include details of the ethical approval gained to carry out the study. For example, the strategies used to gain participants’ consent to take part in the study. The authors should make clear if any ethical issues arose and how these were resolved or managed.

The approach to data analysis (see ref  10 ) needs to be clearly articulated, for example, was there more than one person responsible for analysing the data? How were any discrepancies in findings resolved? An audit trail of how the data were analysed including its management should be documented. If member checking was used this should also be reported. This level of transparency contributes to the trustworthiness and credibility of qualitative research. Some researchers provide a diagram of how they approached data analysis to demonstrate the rigour applied ( figure 1 ).

• Download figure
• Open in new tab
• Download powerpoint

Example of data analysis diagram.

Validity and rigour

The study’s validity is reliant on the statement of the question/problem, theoretical/conceptual framework, design, method, sample and data analysis. When critiquing qualitative research, these elements will help you to determine the study’s reliability. Noble and Smith 11 explain that validity is the integrity of data methods applied and that findings should accurately reflect the data. Rigour should acknowledge the researcher’s role and involvement as well as any biases. Essentially it should focus on truth value, consistency and neutrality and applicability. 11 The authors should discuss if they used triangulation (see table 2 ) to develop the best possible understanding of the phenomena.

Themes and interpretations and implications for practice

In qualitative research no hypothesis is tested, therefore, there is no specific result. Instead, qualitative findings are often reported in themes based on the data analysed. The findings should be clearly linked to, and reflect, the data. This contributes to the soundness of the research. 11 The researchers should make it clear how they arrived at the interpretations of the findings. The theoretical or conceptual framework used should be discussed aiding the rigour of the study. The implications of the findings need to be made clear and where appropriate their applicability or transferability should be identified. 12

Discussions, recommendations and conclusions

The discussion should relate to the research findings as the authors seek to make connections with the literature reviewed earlier in the paper to contextualise their work. A strong discussion will connect the research aims and objectives to the findings and will be supported with literature if possible. A paper that seeks to influence nursing practice will have a recommendations section for clinical practice and research. A good conclusion will focus on the findings and discussion of the phenomena investigated.

Qualitative research has much to offer nursing and healthcare, in terms of understanding patients’ experience of illness, treatment and recovery, it can also help to understand better areas of healthcare practice. However, it must be done with rigour and this paper provides some guidance for appraising such research. To help you critique a qualitative research paper some guidance is provided in table 4 .

Some guidance for critiquing qualitative research

• ↵ Nursing and Midwifery Council . The code: Standard of conduct, performance and ethics for nurses and midwives . 2015 https://www.nmc.org.uk/globalassets/sitedocuments/nmc-publications/nmc-code.pdf ( accessed 21 Aug 18 ).
• Barrett D ,
• Cathala X ,
• Shorten A ,

Patient consent for publication Not required.

Competing interests None declared.

Provenance and peer review Commissioned; internally peer reviewed.

Read the full text or download the PDF:

Critical Appraisal: Assessing the Quality of Studies

• First Online: 05 August 2020

Cite this chapter

• Edward Purssell   ORCID: orcid.org/0000-0003-3748-0864 3 &
• Niall McCrae   ORCID: orcid.org/0000-0001-9776-7694 4  

8573 Accesses

There is great variation in the type and quality of research evidence. Having completed your search and assembled your studies, the next step is to critically appraise the studies to ascertain their quality. Ultimately you will be making a judgement about the overall evidence, but that comes later. You will see throughout this chapter that we make a clear differentiation between the individual studies and what we call the body of evidence , which is all of the studies and anything else that we use to answer the question or to make a recommendation. This chapter deals with only the first of these—the individual studies. Critical appraisal, like everything else in systematic literature reviewing, is a scientific exercise that requires individual judgement, and we describe some tools to help you.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Oxford Centre for Evidence-Based Medicine (OCEBM) (2016) OCEBM levels of evidence. In: CEBM. https://www.cebm.net/2016/05/ocebm-levels-of-evidence/ . Accessed 17 Apr 2020

Aromataris E, Munn Z (eds) (2017) Joanna Briggs Institute reviewer’s manual. The Joanna Briggs Institute, Adelaide

Google Scholar  

Daly J, Willis K, Small R et al (2007) A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol 60:43–49. https://doi.org/10.1016/j.jclinepi.2006.03.014

Article   PubMed   Google Scholar  

EQUATOR Network (2020) What is a reporting guideline?—The EQUATOR Network. https://www.equator-network.org/about-us/what-is-a-reporting-guideline/ . Accessed 7 Mar 2020

Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 19:349–357. https://doi.org/10.1093/intqhc/mzm042

von Elm E, Altman DG, Egger M et al (2007) The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med 4:e296. https://doi.org/10.1371/journal.pmed.0040296

Article   Google Scholar  

Brouwers MC, Kerkvliet K, Spithoff K, AGREE Next Steps Consortium (2016) The AGREE reporting checklist: a tool to improve reporting of clinical practice guidelines. BMJ 352:i1152. https://doi.org/10.1136/bmj.i1152

Article   PubMed   PubMed Central   Google Scholar  

Moher D, Liberati A, Tetzlaff J et al (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6:e1000097. https://doi.org/10.1371/journal.pmed.1000097

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A (2019) Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019), Cochrane. https://www.training.cochrane.org/handbook

Critical Appraisal Skills Programme (2018) CASP checklists. In: CASP—critical appraisal skills programme. https://casp-uk.net/casp-tools-checklists/ . Accessed 7 Mar 2020

Higgins JPT, Savović J, Page MJ et al (2019) Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Chapter   Google Scholar  

Guyatt GH, Oxman AD, Kunz R et al (2011) GRADE guidelines 6. Rating the quality of evidence—imprecision. J Clin Epidemiol 64:1283–1293. https://doi.org/10.1016/j.jclinepi.2011.01.012

Sterne JAC, Savović J, Page MJ et al (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. https://doi.org/10.1136/bmj.l4898

Sterne JA, Hernán MA, Reeves BC et al (2016) ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919. https://doi.org/10.1136/bmj.i4919

Wells GA, Shea B, O’Connell D et al (2019) The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute, Ottawa. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp . Accessed 7 Mar 2020

Cochrane Community (2020) Glossary—Cochrane community. https://community.cochrane.org/glossary#letter-R . Accessed 8 Mar 2020

Messick S (1989) Meaning and values in test validation: the science and ethics of assessment. Educ Res 18:5–11. https://doi.org/10.3102/0013189X018002005

Sparkes AC (2001) Myth 94: qualitative health researchers will agree about validity. Qual Health Res 11:538–552. https://doi.org/10.1177/104973230101100409

Article   CAS   PubMed   Google Scholar  

Aguinis H, Solarino AM (2019) Transparency and replicability in qualitative research: the case of interviews with elite informants. Strat Manag J 40:1291–1315. https://doi.org/10.1002/smj.3015

Lincoln YS, Guba EG (1985) Naturalistic inquiry. Sage Publications, Beverly Hills, CA

Book   Google Scholar  

Hannes K (2011) Chapter 4: Critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K et al (eds) Supplementary guidance for inclusion of qualitative research in Cochrane systematic reviews of interventions. Cochrane Collaboration Qualitative Methods Group, London

Munn Z, Porritt K, Lockwood C et al (2014) Establishing confidence in the output of qualitative research synthesis: the ConQual approach. BMC Med Res Methodol 14:108. https://doi.org/10.1186/1471-2288-14-108

Toye F, Seers K, Allcock N et al (2013) ‘Trying to pin down jelly’—exploring intuitive processes in quality assessment for meta-ethnography. BMC Med Res Methodol 13:46. https://doi.org/10.1186/1471-2288-13-46

Katikireddi SV, Egan M, Petticrew M (2015) How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health 69:189–195. https://doi.org/10.1136/jech-2014-204711

McKenzie JE, Brennan SE, Ryan RE et al (2019) Chapter 9: Summarizing study characteristics and preparing for synthesis. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Deeks JJ, Higgins JPT, Altman DG (2019) Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Download references

Author information

Authors and affiliations.

School of Health Sciences, City, University of London, London, UK

Edward Purssell

Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care, King’s College London, London, UK

Niall McCrae

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Edward Purssell .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Purssell, E., McCrae, N. (2020). Critical Appraisal: Assessing the Quality of Studies. In: How to Perform a Systematic Literature Review. Springer, Cham. https://doi.org/10.1007/978-3-030-49672-2_6

Download citation

DOI : https://doi.org/10.1007/978-3-030-49672-2_6

Published : 05 August 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-49671-5

Online ISBN : 978-3-030-49672-2

eBook Packages : Medicine Medicine (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Am J Pharm Educ
• v.74(8); 2010 Oct 11

Presenting and Evaluating Qualitative Research

The purpose of this paper is to help authors to think about ways to present qualitative research papers in the American Journal of Pharmaceutical Education . It also discusses methods for reviewers to assess the rigour, quality, and usefulness of qualitative research. Examples of different ways to present data from interviews, observations, and focus groups are included. The paper concludes with guidance for publishing qualitative research and a checklist for authors and reviewers.

INTRODUCTION

Policy and practice decisions, including those in education, increasingly are informed by findings from qualitative as well as quantitative research. Qualitative research is useful to policymakers because it often describes the settings in which policies will be implemented. Qualitative research is also useful to both pharmacy practitioners and pharmacy academics who are involved in researching educational issues in both universities and practice and in developing teaching and learning.

Qualitative research involves the collection, analysis, and interpretation of data that are not easily reduced to numbers. These data relate to the social world and the concepts and behaviors of people within it. Qualitative research can be found in all social sciences and in the applied fields that derive from them, for example, research in health services, nursing, and pharmacy. 1 It looks at X in terms of how X varies in different circumstances rather than how big is X or how many Xs are there? 2 Textbooks often subdivide research into qualitative and quantitative approaches, furthering the common assumption that there are fundamental differences between the 2 approaches. With pharmacy educators who have been trained in the natural and clinical sciences, there is often a tendency to embrace quantitative research, perhaps due to familiarity. A growing consensus is emerging that sees both qualitative and quantitative approaches as useful to answering research questions and understanding the world. Increasingly mixed methods research is being carried out where the researcher explicitly combines the quantitative and qualitative aspects of the study. 3 , 4

Like healthcare, education involves complex human interactions that can rarely be studied or explained in simple terms. Complex educational situations demand complex understanding; thus, the scope of educational research can be extended by the use of qualitative methods. Qualitative research can sometimes provide a better understanding of the nature of educational problems and thus add to insights into teaching and learning in a number of contexts. For example, at the University of Nottingham, we conducted in-depth interviews with pharmacists to determine their perceptions of continuing professional development and who had influenced their learning. We also have used a case study approach using observation of practice and in-depth interviews to explore physiotherapists' views of influences on their leaning in practice. We have conducted in-depth interviews with a variety of stakeholders in Malawi, Africa, to explore the issues surrounding pharmacy academic capacity building. A colleague has interviewed and conducted focus groups with students to explore cultural issues as part of a joint Nottingham-Malaysia pharmacy degree program. Another colleague has interviewed pharmacists and patients regarding their expectations before and after clinic appointments and then observed pharmacist-patient communication in clinics and assessed it using the Calgary Cambridge model in order to develop recommendations for communication skills training. 5 We have also performed documentary analysis on curriculum data to compare pharmacist and nurse supplementary prescribing courses in the United Kingdom.

It is important to choose the most appropriate methods for what is being investigated. Qualitative research is not appropriate to answer every research question and researchers need to think carefully about their objectives. Do they wish to study a particular phenomenon in depth (eg, students' perceptions of studying in a different culture)? Or are they more interested in making standardized comparisons and accounting for variance (eg, examining differences in examination grades after changing the way the content of a module is taught). Clearly a quantitative approach would be more appropriate in the last example. As with any research project, a clear research objective has to be identified to know which methods should be applied.

Types of qualitative data include:

• Audio recordings and transcripts from in-depth or semi-structured interviews
• Structured interview questionnaires containing substantial open comments including a substantial number of responses to open comment items.
• Audio recordings and transcripts from focus group sessions.
• Field notes (notes taken by the researcher while in the field [setting] being studied)
• Video recordings (eg, lecture delivery, class assignments, laboratory performance)
• Case study notes
• Documents (reports, meeting minutes, e-mails)
• Diaries, video diaries
• Observation notes
• Press clippings
• Photographs

RIGOUR IN QUALITATIVE RESEARCH

Qualitative research is often criticized as biased, small scale, anecdotal, and/or lacking rigor; however, when it is carried out properly it is unbiased, in depth, valid, reliable, credible and rigorous. In qualitative research, there needs to be a way of assessing the “extent to which claims are supported by convincing evidence.” 1 Although the terms reliability and validity traditionally have been associated with quantitative research, increasingly they are being seen as important concepts in qualitative research as well. Examining the data for reliability and validity assesses both the objectivity and credibility of the research. Validity relates to the honesty and genuineness of the research data, while reliability relates to the reproducibility and stability of the data.

The validity of research findings refers to the extent to which the findings are an accurate representation of the phenomena they are intended to represent. The reliability of a study refers to the reproducibility of the findings. Validity can be substantiated by a number of techniques including triangulation use of contradictory evidence, respondent validation, and constant comparison. Triangulation is using 2 or more methods to study the same phenomenon. Contradictory evidence, often known as deviant cases, must be sought out, examined, and accounted for in the analysis to ensure that researcher bias does not interfere with or alter their perception of the data and any insights offered. Respondent validation, which is allowing participants to read through the data and analyses and provide feedback on the researchers' interpretations of their responses, provides researchers with a method of checking for inconsistencies, challenges the researchers' assumptions, and provides them with an opportunity to re-analyze their data. The use of constant comparison means that one piece of data (for example, an interview) is compared with previous data and not considered on its own, enabling researchers to treat the data as a whole rather than fragmenting it. Constant comparison also enables the researcher to identify emerging/unanticipated themes within the research project.

STRENGTHS AND LIMITATIONS OF QUALITATIVE RESEARCH

Qualitative researchers have been criticized for overusing interviews and focus groups at the expense of other methods such as ethnography, observation, documentary analysis, case studies, and conversational analysis. Qualitative research has numerous strengths when properly conducted.

Strengths of Qualitative Research

• Issues can be examined in detail and in depth.
• Interviews are not restricted to specific questions and can be guided/redirected by the researcher in real time.
• The research framework and direction can be quickly revised as new information emerges.
• The data based on human experience that is obtained is powerful and sometimes more compelling than quantitative data.
• Subtleties and complexities about the research subjects and/or topic are discovered that are often missed by more positivistic enquiries.
• Data usually are collected from a few cases or individuals so findings cannot be generalized to a larger population. Findings can however be transferable to another setting.

Limitations of Qualitative Research

• Research quality is heavily dependent on the individual skills of the researcher and more easily influenced by the researcher's personal biases and idiosyncrasies.
• Rigor is more difficult to maintain, assess, and demonstrate.
• The volume of data makes analysis and interpretation time consuming.
• It is sometimes not as well understood and accepted as quantitative research within the scientific community
• The researcher's presence during data gathering, which is often unavoidable in qualitative research, can affect the subjects' responses.
• Issues of anonymity and confidentiality can present problems when presenting findings
• Findings can be more difficult and time consuming to characterize in a visual way.

PRESENTATION OF QUALITATIVE RESEARCH FINDINGS

The following extracts are examples of how qualitative data might be presented:

Data From an Interview.

The following is an example of how to present and discuss a quote from an interview.

The researcher should select quotes that are poignant and/or most representative of the research findings. Including large portions of an interview in a research paper is not necessary and often tedious for the reader. The setting and speakers should be established in the text at the end of the quote.

The student describes how he had used deep learning in a dispensing module. He was able to draw on learning from a previous module, “I found that while using the e learning programme I was able to apply the knowledge and skills that I had gained in last year's diseases and goals of treatment module.” (interviewee 22, male)

This is an excerpt from an article on curriculum reform that used interviews 5 :

The first question was, “Without the accreditation mandate, how much of this curriculum reform would have been attempted?” According to respondents, accreditation played a significant role in prompting the broad-based curricular change, and their comments revealed a nuanced view. Most indicated that the change would likely have occurred even without the mandate from the accreditation process: “It reflects where the profession wants to be … training a professional who wants to take on more responsibility.” However, they also commented that “if it were not mandated, it could have been a very difficult road.” Or it “would have happened, but much later.” The change would more likely have been incremental, “evolutionary,” or far more limited in its scope. “Accreditation tipped the balance” was the way one person phrased it. “Nobody got serious until the accrediting body said it would no longer accredit programs that did not change.”

Data From Observations

The following example is some data taken from observation of pharmacist patient consultations using the Calgary Cambridge guide. 6 , 7 The data are first presented and a discussion follows:

Pharmacist: We will soon be starting a stop smoking clinic. Patient: Is the interview over now? Pharmacist: No this is part of it. (Laughs) You can't tell me to bog off (sic) yet. (pause) We will be starting a stop smoking service here, Patient: Yes. Pharmacist: with one-to-one and we will be able to help you or try to help you. If you want it. In this example, the pharmacist has picked up from the patient's reaction to the stop smoking clinic that she is not receptive to advice about giving up smoking at this time; in fact she would rather end the consultation. The pharmacist draws on his prior relationship with the patient and makes use of a joke to lighten the tone. He feels his message is important enough to persevere but he presents the information in a succinct and non-pressurised way. His final comment of “If you want it” is important as this makes it clear that he is not putting any pressure on the patient to take up this offer. This extract shows that some patient cues were picked up, and appropriately dealt with, but this was not the case in all examples.

Data From Focus Groups

This excerpt from a study involving 11 focus groups illustrates how findings are presented using representative quotes from focus group participants. 8

Those pharmacists who were initially familiar with CPD endorsed the model for their peers, and suggested it had made a meaningful difference in the way they viewed their own practice. In virtually all focus groups sessions, pharmacists familiar with and supportive of the CPD paradigm had worked in collaborative practice environments such as hospital pharmacy practice. For these pharmacists, the major advantage of CPD was the linking of workplace learning with continuous education. One pharmacist stated, “It's amazing how much I have to learn every day, when I work as a pharmacist. With [the learning portfolio] it helps to show how much learning we all do, every day. It's kind of satisfying to look it over and see how much you accomplish.” Within many of the learning portfolio-sharing sessions, debates emerged regarding the true value of traditional continuing education and its outcome in changing an individual's practice. While participants appreciated the opportunity for social and professional networking inherent in some forms of traditional CE, most eventually conceded that the academic value of most CE programming was limited by the lack of a systematic process for following-up and implementing new learning in the workplace. “Well it's nice to go to these [continuing education] events, but really, I don't know how useful they are. You go, you sit, you listen, but then, well I at least forget.”

The following is an extract from a focus group (conducted by the author) with first-year pharmacy students about community placements. It illustrates how focus groups provide a chance for participants to discuss issues on which they might disagree.

Interviewer: So you are saying that you would prefer health related placements? Student 1: Not exactly so long as I could be developing my communication skill. Student 2: Yes but I still think the more health related the placement is the more I'll gain from it. Student 3: I disagree because other people related skills are useful and you may learn those from taking part in a community project like building a garden. Interviewer: So would you prefer a mixture of health and non health related community placements?

GUIDANCE FOR PUBLISHING QUALITATIVE RESEARCH

Qualitative research is becoming increasingly accepted and published in pharmacy and medical journals. Some journals and publishers have guidelines for presenting qualitative research, for example, the British Medical Journal 9 and Biomedcentral . 10 Medical Education published a useful series of articles on qualitative research. 11 Some of the important issues that should be considered by authors, reviewers and editors when publishing qualitative research are discussed below.

Introduction.

A good introduction provides a brief overview of the manuscript, including the research question and a statement justifying the research question and the reasons for using qualitative research methods. This section also should provide background information, including relevant literature from pharmacy, medicine, and other health professions, as well as literature from the field of education that addresses similar issues. Any specific educational or research terminology used in the manuscript should be defined in the introduction.

The methods section should clearly state and justify why the particular method, for example, face to face semistructured interviews, was chosen. The method should be outlined and illustrated with examples such as the interview questions, focusing exercises, observation criteria, etc. The criteria for selecting the study participants should then be explained and justified. The way in which the participants were recruited and by whom also must be stated. A brief explanation/description should be included of those who were invited to participate but chose not to. It is important to consider “fair dealing,” ie, whether the research design explicitly incorporates a wide range of different perspectives so that the viewpoint of 1 group is never presented as if it represents the sole truth about any situation. The process by which ethical and or research/institutional governance approval was obtained should be described and cited.

The study sample and the research setting should be described. Sampling differs between qualitative and quantitative studies. In quantitative survey studies, it is important to select probability samples so that statistics can be used to provide generalizations to the population from which the sample was drawn. Qualitative research necessitates having a small sample because of the detailed and intensive work required for the study. So sample sizes are not calculated using mathematical rules and probability statistics are not applied. Instead qualitative researchers should describe their sample in terms of characteristics and relevance to the wider population. Purposive sampling is common in qualitative research. Particular individuals are chosen with characteristics relevant to the study who are thought will be most informative. Purposive sampling also may be used to produce maximum variation within a sample. Participants being chosen based for example, on year of study, gender, place of work, etc. Representative samples also may be used, for example, 20 students from each of 6 schools of pharmacy. Convenience samples involve the researcher choosing those who are either most accessible or most willing to take part. This may be fine for exploratory studies; however, this form of sampling may be biased and unrepresentative of the population in question. Theoretical sampling uses insights gained from previous research to inform sample selection for a new study. The method for gaining informed consent from the participants should be described, as well as how anonymity and confidentiality of subjects were guaranteed. The method of recording, eg, audio or video recording, should be noted, along with procedures used for transcribing the data.

Data Analysis.

A description of how the data were analyzed also should be included. Was computer-aided qualitative data analysis software such as NVivo (QSR International, Cambridge, MA) used? Arrival at “data saturation” or the end of data collection should then be described and justified. A good rule when considering how much information to include is that readers should have been given enough information to be able to carry out similar research themselves.

One of the strengths of qualitative research is the recognition that data must always be understood in relation to the context of their production. 1 The analytical approach taken should be described in detail and theoretically justified in light of the research question. If the analysis was repeated by more than 1 researcher to ensure reliability or trustworthiness, this should be stated and methods of resolving any disagreements clearly described. Some researchers ask participants to check the data. If this was done, it should be fully discussed in the paper.

An adequate account of how the findings were produced should be included A description of how the themes and concepts were derived from the data also should be included. Was an inductive or deductive process used? The analysis should not be limited to just those issues that the researcher thinks are important, anticipated themes, but also consider issues that participants raised, ie, emergent themes. Qualitative researchers must be open regarding the data analysis and provide evidence of their thinking, for example, were alternative explanations for the data considered and dismissed, and if so, why were they dismissed? It also is important to present outlying or negative/deviant cases that did not fit with the central interpretation.

The interpretation should usually be grounded in interviewees or respondents' contributions and may be semi-quantified, if this is possible or appropriate, for example, “Half of the respondents said …” “The majority said …” “Three said…” Readers should be presented with data that enable them to “see what the researcher is talking about.” 1 Sufficient data should be presented to allow the reader to clearly see the relationship between the data and the interpretation of the data. Qualitative data conventionally are presented by using illustrative quotes. Quotes are “raw data” and should be compiled and analyzed, not just listed. There should be an explanation of how the quotes were chosen and how they are labeled. For example, have pseudonyms been given to each respondent or are the respondents identified using codes, and if so, how? It is important for the reader to be able to see that a range of participants have contributed to the data and that not all the quotes are drawn from 1 or 2 individuals. There is a tendency for authors to overuse quotes and for papers to be dominated by a series of long quotes with little analysis or discussion. This should be avoided.

Participants do not always state the truth and may say what they think the interviewer wishes to hear. A good qualitative researcher should not only examine what people say but also consider how they structured their responses and how they talked about the subject being discussed, for example, the person's emotions, tone, nonverbal communication, etc. If the research was triangulated with other qualitative or quantitative data, this should be discussed.

Discussion.

The findings should be presented in the context of any similar previous research and or theories. A discussion of the existing literature and how this present research contributes to the area should be included. A consideration must also be made about how transferrable the research would be to other settings. Any particular strengths and limitations of the research also should be discussed. It is common practice to include some discussion within the results section of qualitative research and follow with a concluding discussion.

The author also should reflect on their own influence on the data, including a consideration of how the researcher(s) may have introduced bias to the results. The researcher should critically examine their own influence on the design and development of the research, as well as on data collection and interpretation of the data, eg, were they an experienced teacher who researched teaching methods? If so, they should discuss how this might have influenced their interpretation of the results.

Conclusion.

The conclusion should summarize the main findings from the study and emphasize what the study adds to knowledge in the area being studied. Mays and Pope suggest the researcher ask the following 3 questions to determine whether the conclusions of a qualitative study are valid 12 : How well does this analysis explain why people behave in the way they do? How comprehensible would this explanation be to a thoughtful participant in the setting? How well does the explanation cohere with what we already know?

CHECKLIST FOR QUALITATIVE PAPERS

This paper establishes criteria for judging the quality of qualitative research. It provides guidance for authors and reviewers to prepare and review qualitative research papers for the American Journal of Pharmaceutical Education . A checklist is provided in Appendix 1 to assist both authors and reviewers of qualitative data.

ACKNOWLEDGEMENTS

Thank you to the 3 reviewers whose ideas helped me to shape this paper.

Appendix 1. Checklist for authors and reviewers of qualitative research.

Introduction

• □ Research question is clearly stated.
• □ Research question is justified and related to the existing knowledge base (empirical research, theory, policy).
• □ Any specific research or educational terminology used later in manuscript is defined.
• □ The process by which ethical and or research/institutional governance approval was obtained is described and cited.
• □ Reason for choosing particular research method is stated.
• □ Criteria for selecting study participants are explained and justified.
• □ Recruitment methods are explicitly stated.
• □ Details of who chose not to participate and why are given.
• □ Study sample and research setting used are described.
• □ Method for gaining informed consent from the participants is described.
• □ Maintenance/Preservation of subject anonymity and confidentiality is described.
• □ Method of recording data (eg, audio or video recording) and procedures for transcribing data are described.
• □ Methods are outlined and examples given (eg, interview guide).
• □ Decision to stop data collection is described and justified.
• □ Data analysis and verification are described, including by whom they were performed.
• □ Methods for identifying/extrapolating themes and concepts from the data are discussed.
• □ Sufficient data are presented to allow a reader to assess whether or not the interpretation is supported by the data.
• □ Outlying or negative/deviant cases that do not fit with the central interpretation are presented.
• □ Transferability of research findings to other settings is discussed.
• □ Findings are presented in the context of any similar previous research and social theories.
• □ Discussion often is incorporated into the results in qualitative papers.
• □ A discussion of the existing literature and how this present research contributes to the area is included.
• □ Any particular strengths and limitations of the research are discussed.
• □ Reflection of the influence of the researcher(s) on the data, including a consideration of how the researcher(s) may have introduced bias to the results is included.

Conclusions

• □ The conclusion states the main finings of the study and emphasizes what the study adds to knowledge in the subject area.

• UNC Libraries
• HSL Subject Research
• Qualitative Research Resources
• Assessing Qualitative Research

Qualitative Research Resources: Assessing Qualitative Research

Created by health science librarians.

• What is Qualitative Research?
• Qualitative Research Basics
• Special Topics
• Training Opportunities: UNC & Beyond
• Help at UNC
• Qualitative Software for Coding/Analysis
• Software for Audio, Video, Online Surveys
• Finding Qualitative Studies

About this Page

Legend (let evidence guide every new decision) assessment tools: cincinnati children's hospital, equator network: enhancing the quality and transparency of health research, other tools for assessing qualitative research.

• Writing Up Your Research
• Integrating Qualitative Research into Systematic Reviews
• Publishing Qualitative Research
• Presenting Qualitative Research
• Qualitative & Libraries: a few gems
• Data Repositories

Why is this information important?

• Qualitative research typically focuses on collecting very detailed information on a few cases and often addresses meaning, rather than objectively identifiable factors.
• This means that typical markers of research quality for quantitative studies, such as validity and reliability, cannot be used to assess qualitative research.

On this page you'll find:

The resources on this page will guide you to some of the alternative measures/tools or means you can use to assess qualitative research.

Evidence Evaluation Tools and Resources

This website has a number of resources for evaluating health sciences research across a variety of designs/study types, including an Evidence Appraisal form for qualitative research (in table), as well as forms for mixed methods studies from a variety of clinical question domains. The site includes information on the following:

• Evaluating the Evidence Algorithm (pdf download)
• Evidence Appraisal Forms ( see Domain of Clinical Questions Table )
• Table of Evidence Levels (pdf download)
• Grading a Body of Evidence (pdf download)
• Judging the Strength of a Recommendation (pdf download)
• LEGEND Glossary (pdf download)
• EQUATOR: Qualitative Research Reporting Guidelines
• EQUATOR Network Home

The EQUATOR Network is an ‘umbrella’ organisation that brings together researchers, medical journal editors, peer reviewers, developers of reporting guidelines, research funding bodies and other collaborators with mutual interest in improving the quality of research publications and of research itself. 

The EQUATOR Library contains a comprehensive searchable database of reporting guidelines for many study types--including qualitative--and also links to other resources relevant to research reporting:

• Library for health research reporting:  provides an up-to-date collection of guidelines and policy documents related to health research reporting. These are aimed mainly at authors of research articles, journal editors, peer reviewers and reporting guideline developers.
• Toolkits to support writing research, using guidelines, teaching research skills, selecting the appropriate reporting guideline
• Courses and events
• Librarian Network

Also see Articles box, below, some of which contain checklists or tools. 

Most checklists or tools are meant to help you think critically and systematically when appraising research.  Users should generally consult accompanying materials such as manuals, handbooks, and cited literature to use these tools appropriately.  Broad understanding of the variety and complexity of qualitative research is generally necessary, along with an understanding of the philosophical perspectives plus knowledge about specific qualitative research methods and their implementation.  

• CASP/Critical Assessment Skills Programme Tool for Evaluating Qualitative Research 2018
• CASP Knowledge Hub Includes critical appraisal checklists for key study designs; glossary of key research terms; key links related to evidence based healthcare, statistics, and research; a bibliography of articles and research papers about CASP and other critical appraisal tools and approaches 1993-2012.
• (Joanna Briggs Institute) Manual for Evidence Synthesis (2024) See the following chapters: Chapter 2: Systematic reviews of qualitative evidence. Includes appendices: • Appendix 2.1: Critical Appraisal Checklist for Qualitative Research • Appendix 2.2: Discussion of Qualitative critical appraisal criteria • Appendix 2.3 Qualitative data extraction tool Chapter 8: Mixed methods systematic reviews more... less... Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://synthesismanual.jbi.global. https://doi.org/10.46658/JBIMES-20-01
• McGill Mixed Methods Appraisal Tool (MMAT) Front Page Public wiki site for the MMAT: The MMAT is intended to be used as a checklist for concomitantly appraising and/or describing studies included in systematic mixed studies reviews (reviews including original qualitative, quantitative and mixed methods studies). The MMAT was first published in 2009. Since then, it has been validated in several studies testing its interrater reliability, usability and content validity. The latest version of the MMAT was updated in 2018.
• McGill Mixed Methods Appraisal Tool (MMAT) 2018 User Guide See full site (public wiki link above) for additional information, including FAQ's, references and resources, earlier versions, and more.
• McMaster University Critical Review Form & Guidelines for Qualitative Studies v2.0 Includes links to Qualitative Review Form (v2.0) and accompanying Guidelines from the Evidence Based Practice Research Group of McMaster University's School of Rehabilitation Science). Links are also provided for Spanish, German, and French versions.
• NICE Quality Appraisal Checklist-Qualitative Studies, 3rd ed, 2012, from UK National Institute for Health and Care Excellence Includes checklist and notes on its use. From Methods for the Development of NICE Public Health Guidance, 3rd edition. more... less... Produced by the National Institute for Health and Clinical Excellence © Copyright National Institute for Health and Clinical Excellence, 2006 (updated 2012). All rights reserved. This material may be freely reproduced for educational and not-for-profit purposes. No reproduction by or for commercial organisations, or for commercial purposes, is allowed without the express written permission of the Institute.
• NICE Quality Appraisal Checklist-Qualitative Studies, 3rd ed. (.pdf download) Appendix H Checklist and Notes download. © Copyright National Institute for Health and Clinical Excellence, 2006 (updated 2012). All rights reserved. This material may be freely reproduced for educational and not-for-profit purposes. No reproduction by or for commercial organisations, or for commercial purposes, is allowed without the express written permission of the Institute.
• Qualitative Research Review Guidelines, RATS
• SBU Swedish Agency for Health Technology Assessment and Assessment of Social Services Evaluation and synthesis of studies using qualitative methods of analysis, 2016. Appendix 2 of this document (at the end) contains a checklist for evaluating qualitative research. more... less... SBU. Evaluation and synthesis of studies using qualitative methods of analysis. Stockholm: Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU); 2016.
• Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice, 3rd ed (JAMA Evidence) Chapter 13.5 Qualitative Research
• Slides: Appraising Qualitative Research from Users' Guide to the Medical Literature, 3rd edition Click on the 'Related Content' tab to find the link to download the Appraising Qualitative Research slides.

These articles address a range of issues related to understanding and evaluating qualitative research; some  include checklists or tools.

Clissett, P. (2008) "Evaluating Qualitative Research." Journal of Orthopaedic Nursing 12: 99-105.

Cohen, Deborah J. and Benjamin F. Crabtree. (2008) "Evidence for Qualitative Research in Health Care: Controversies and Recommendations." Annals of Family Medicine 6(4): 331-339.

• Supplemental Appendix 1. Search Strategy for Criteria for Qualitative Research in Health Care
• Supplemental Appendix 2. Publications Analyzed: Health Care Journals and Frequently Referenced Books and Book Chapters (1980-2005) That Posited Criteria for "Good" Qualitative Research.

Dixon-Woods, M.,  R.L. Shaw, S. Agarwal, and J.A. Smith. (2004) "The Problem of Appraising Qualitative Research." Qual Safe Health Care 13: 223-225.

Fossey, E., C. Harvey, F. McDermott, and L. Davidson. (2002) "Understanding and Evaluating Qualitative Research." Australian and New Zealand Journal of Psychiatry 36(6): 717-732.

Hammarberg, K., M. Kirkman, S. de Lacey. (2016) "Qualitative Research Methods: When to Use and How to Judge them." Human Reproduction 31 (3): 498-501.

Lee, J. (2014) "Genre-Appropriate Judgments of Qualitative Research." Philosophy of the Social Sciences 44(3): 316-348. (This provides 3 strategies for evaluating qualitative research, 2 that the author is not crazy about and one that he considers more appropriate/accurate).

Majid, Umair and Vanstone,Meredith (2018). "Appraising Qualitative Research for Evidence Syntheses: A Compendium of Quality Appraisal Tools." Qualitative Health Research  28(13): 2115-2131.   PMID: 30047306 DOI:  10.1177/1049732318785358

Meyrick, Jane. (2006) "What is Good Qualitative Research? A First Step towards a Comprehensive Approach to Judging Rigour/Quality." Journal of Health Psychology 11(5): 799-808.

Miles, MB, AM Huberman, J Saldana. (2014) Qualitative Data Analysis.  Thousand Oaks, Califorinia, SAGE Publications, Inc. Chapter 11: Drawing and Verifying Conclusions . Check Availability of Print Book . 

Morse, JM. (1997) "Perfectly Healthy but Dead:"The Myth of Inter-Rater Reliability. Qualitative Health Research 7(4): 445-447.  

O’Brien BC, Harris IB, Beckman TJ, et al. (2014) Standards for reporting qualitative research: a synthesis of recommendations . Acad Med 89(9):1245–1251. DOI: 10.1097/ACM.0000000000000388 PMID: 24979285

The Standards for Reporting Qualitative Research (SRQR) consists of 21 items. The authors define and explain key elements of each item and provide examples from recently published articles to illustrate ways in which the standards can be met. The SRQR aims to improve the transparency of all aspects of qualitative research by providing clear standards for reporting qualitative research. These standards will assist authors during manuscript preparation, editors and reviewers in evaluating a manuscript for potential publication, and readers when critically appraising, applying, and synthesizing study findings.

Ryan, Frances, Michael Coughlin, and Patricia Cronin. (2007) "Step by Step Guide to Critiquing Research: Part 2, Qualitative Research." British Journal of Nursing 16(12): 738-744.

Stige, B, K. Malterud, and T. Midtgarden. (2009) "Toward an Agenda for Evaluation of Qualitative Research." Qualitative Health Research 19(10): 1504-1516.

Tong, Allison and Mary Amanda Dew. (2016-EPub ahead of print). "Qualitative Research in Transplantation: Ensuring Relevance and Rigor. "   Transplantation 

Allison Tong, Peter Sainsbury, Jonathan Craig; Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups , International Journal for Quality in Health Care , Volume 19, Issue 6, 1 December 2007, Pages 349–357, https://doi.org/10.1093/intqhc/mzm042

The criteria included in COREQ, a 32-item checklist, can help researchers to report important aspects of the research team, study methods, context of the study, findings, analysis and interpretations. Items most frequently included in the checklists related to sampling method, setting for data collection, method of data collection, respondent validation of findings, method of recording data, description of the derivation of themes and inclusion of supporting quotations. We grouped all items into three domains: (i) research team and reflexivity, (ii) study design and (iii) data analysis and reporting.

Tracy, Sarah (2010) “Qualitative Quality: Eight ‘Big-Tent’ Criteria for Excellent Qualitative Research.” Qualitative Inquiry 16(10):837-51

• Critical Appraisal Skills Programme
• IMPSCI (Implementation Science) Tutorials
• Johns Hopkins: Why Mixed Methods?
• Measuring, Learning, and Evaluation Project for the Urban Reproductive Health Initiative This project ran 2010-2015. Some project resources are still available.
• NIH OBSSR (Office of Behavioral & Social Sciences Research) Best Practices for Mixed Methods Research in Health Sciences, 2011 The OBSSR commissioned a team in 2010 to develop a resource that would provide guidance to NIH investigators on how to rigorously develop and evaluate mixed methods research applications. more... less... John W. Creswell, Ph.D., University of Nebraska-Lincoln Ann Carroll Klassen, Ph.D., Drexel University Vicki L. Plano Clark, Ph.D., University of Nebraska-Lincoln Katherine Clegg Smith, Ph.D., Johns Hopkins University With the Assistance of a Specially Appointed Working Group
• NIH OBSSR eSource: Introductory Social and Behavioral Science Training Materials eSource is a collection of online chapters that provide an introduction to selected behavioral and social science research approaches, including theory development and testing, survey methods, measurement, and study design. more... less... Link not working on OBSSR website, here https://obssr.od.nih.gov/about-us/publications/ Formerly: https://obssr-archive.od.nih.gov/pdf/Qualitative.PDF
• NSF Workshop on Interdisciplinary Standards for Systematic Qualitative Research On May 19-20, 2005, a workshop on Interdisciplinary Standards for Systematic Qualitative Research was held at the National Science Foundation (NSF) in Arlington, Virginia. The workshop was cofunded by a grant from four NSF Programs—Cultural Anthropology, Law and Social Science, Political Science, and Sociology… It is well recognized that each of the four disciplines have different research design and evaluation cultures as well as considerable variability in the emphasis on interpretation and explanation, commitment to constructivist and positivist epistemologies, and the degree of perceived consensus about the value and prominence of qualitative research methods. more... less... Within this multidisciplinary and multimethods context, twenty-four scholars from the four disciplines were charged to (1) articulate the standards used in their particular field to ensure rigor across the range of qualitative methodological approaches;1* (2) identify common criteria shared across the four disciplines for designing and evaluating research proposals and fostering multidisciplinary collaborations; and (3) develop an agenda for strengthening the tools, training, data, research design, and infrastructure for research using qualitative approaches.
• Technical Note: Mixed-Methods Evaluations (USAID) This open source resource from USAID (2013) discusses the mixing of qualitative and quantitative methods in mixed methods research.
• Qualitative Research Methods: A Data Collector's Field Guide (2005) From FHI 360/Family Health International with support from US AID. Natasha Mack, Cynthia Woodsong, Kathleen M. MacQueen, Greg Guest, and Emily Name. The guide is divided into five modules covering the following topics: Module 1 – Qualitative Research Methods Overview Module 2 – Participant Observation Module 3 – In-Depth Interviews Module 4 – Focus Groups Module 5 – Data Documentation and Management
• Robert Wood Johnson Foundation Guidelines for Designing, Analyzing, and Reporting Qualitative Research
• Robert Wood Johnson Foundation: Qualitative Research Guidelines Project

Qualitative Literacy

Not a checklist, this is a thorough discussion of assessing the scientific merit of a study based on in-depth interviews or participant observation, first by assessing exposure (e.g. time exposed in the field). Then, assuming sufficient exposure, the authors propose looking for signs of

• cognitive empathy, how those interviewed or observed perceive themselves and their social world, the meaning they attach to those perceptions, the motives they express for their actions 
• palpability: the evidence would be palpable or concrete rather than abstract or general
• heterogeneity: showing diversity across people, over time, among situations, or between contexts 
• follow-up: responding to the unexpected; following up on unanticipated statements or observations 
• self-awareness: showing that the author is explicitly aware of the impact of their presence on who was accessed and what they disclosed
• << Previous: Finding Qualitative Studies
• Next: Writing Up Your Research >>
• Last Updated: Jul 28, 2024 4:11 PM
• URL: https://guides.lib.unc.edu/qual

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals

You are here

• Volume 27, Issue Suppl 2
• 12 Critical appraisal tools for qualitative research – towards ‘fit for purpose’
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Veronika Williams 1 ,
• Anne-Marie Boylan 2 ,
• Newhouse Nikki 2 ,
• David Nunan 2
• 1 Nipissing University, North Bay, Canada
• 2 University of Oxford, Oxford, UK

Qualitative research has an important place within evidence-based health care (EBHC), contributing to policy on patient safety and quality of care, supporting understanding of the impact of chronic illness, and explaining contextual factors surrounding the implementation of interventions. However, the question of whether, when and how to critically appraise qualitative research persists. Whilst there is consensus that we cannot - and should not – simplistically adopt existing approaches for appraising quantitative methods, it is nonetheless crucial that we develop a better understanding of how to subject qualitative evidence to robust and systematic scrutiny in order to assess its trustworthiness and credibility. Currently, most appraisal methods and tools for qualitative health research use one of two approaches: checklists or frameworks. We have previously outlined the specific issues with these approaches (Williams et al 2019). A fundamental challenge still to be addressed, however, is the lack of differentiation between different methodological approaches when appraising qualitative health research. We do this routinely when appraising quantitative research: we have specific checklists and tools to appraise randomised controlled trials, diagnostic studies, observational studies and so on. Current checklists for qualitative research typically treat the entire paradigm as a single design (illustrated by titles of tools such as ‘CASP Qualitative Checklist’, ‘JBI checklist for qualitative research’) and frameworks tend to require substantial understanding of a given methodological approach without providing guidance on how they should be applied. Given the fundamental differences in the aims and outcomes of different methodologies, such as ethnography, grounded theory, and phenomenological approaches, as well as specific aspects of the research process, such as sampling, data collection and analysis, we cannot treat qualitative research as a single approach. Rather, we must strive to recognise core commonalities relating to rigour, but considering key methodological differences. We have argued for a reconsideration of current approaches to the systematic appraisal of qualitative health research (Williams et al 2021), and propose the development of a tool or tools that allow differentiated evaluations of multiple methodological approaches rather than continuing to treat qualitative health research as a single, unified method. Here we propose a workshop for researchers interested in the appraisal of qualitative health research and invite them to develop an initial consensus regarding core aspects of a new appraisal tool that differentiates between the different qualitative research methodologies and thus provides a ‘fit for purpose’ tool, for both, educators and clinicians.

https://doi.org/10.1136/ebm-2022-EBMLive.36

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Read the full text or download the PDF:

• The Open University
• Accessibility hub
• Guest user / Sign out
• Study with The Open University

My OpenLearn Profile

Personalise your OpenLearn profile, save your favourite content and get recognition for your learning

About this free course

Become an ou student, download this course, share this free course.

Start this free course now. Just create an account and sign in. Enrol and complete the course for a free statement of participation or digital badge if available.

1 Important points to consider when critically evaluating published research papers

Simple review articles (also referred to as ‘narrative’ or ‘selective’ reviews), systematic reviews and meta-analyses provide rapid overviews and ‘snapshots’ of progress made within a field, summarising a given topic or research area. They can serve as useful guides, or as current and comprehensive ‘sources’ of information, and can act as a point of reference to relevant primary research studies within a given scientific area. Narrative or systematic reviews are often used as a first step towards a more detailed investigation of a topic or a specific enquiry (a hypothesis or research question), or to establish critical awareness of a rapidly-moving field (you will be required to demonstrate this as part of an assignment, an essay or a dissertation at postgraduate level).

The majority of primary ‘empirical’ research papers essentially follow the same structure (abbreviated here as IMRAD). There is a section on Introduction, followed by the Methods, then the Results, which includes figures and tables showing data described in the paper, and a Discussion. The paper typically ends with a Conclusion, and References and Acknowledgements sections.

The Title of the paper provides a concise first impression. The Abstract follows the basic structure of the extended article. It provides an ‘accessible’ and concise summary of the aims, methods, results and conclusions. The Introduction provides useful background information and context, and typically outlines the aims and objectives of the study. The Abstract can serve as a useful summary of the paper, presenting the purpose, scope and major findings. However, simply reading the abstract alone is not a substitute for critically reading the whole article. To really get a good understanding and to be able to critically evaluate a research study, it is necessary to read on.

While most research papers follow the above format, variations do exist. For example, the results and discussion sections may be combined. In some journals the materials and methods may follow the discussion, and in two of the most widely read journals, Science and Nature, the format does vary from the above due to restrictions on the length of articles. In addition, there may be supporting documents that accompany a paper, including supplementary materials such as supporting data, tables, figures, videos and so on. There may also be commentaries or editorials associated with a topical research paper, which provide an overview or critique of the study being presented.

Box 1 Key questions to ask when appraising a research paper

• Is the study’s research question relevant?
• Does the study add anything new to current knowledge and understanding?
• Does the study test a stated hypothesis?
• Is the design of the study appropriate to the research question?
• Do the study methods address key potential sources of bias?
• Were suitable ‘controls’ included in the study?
• Were the statistical analyses appropriate and applied correctly?
• Is there a clear statement of findings?
• Does the data support the authors’ conclusions?
• Are there any conflicts of interest or ethical concerns?

There are various strategies used in reading a scientific research paper, and one of these is to start with the title and the abstract, then look at the figures and tables, and move on to the introduction, before turning to the results and discussion, and finally, interrogating the methods.

Another strategy (outlined below) is to begin with the abstract and then the discussion, take a look at the methods, and then the results section (including any relevant tables and figures), before moving on to look more closely at the discussion and, finally, the conclusion. You should choose a strategy that works best for you. However, asking the ‘right’ questions is a central feature of critical appraisal, as with any enquiry, so where should you begin? Here are some critical questions to consider when evaluating a research paper.

Look at the Abstract and then the Discussion : Are these accessible and of general relevance or are they detailed, with far-reaching conclusions? Is it clear why the study was undertaken? Why are the conclusions important? Does the study add anything new to current knowledge and understanding? The reasons why a particular study design or statistical method were chosen should also be clear from reading a research paper. What is the research question being asked? Does the study test a stated hypothesis? Is the design of the study appropriate to the research question? Have the authors considered the limitations of their study and have they discussed these in context?

Take a look at the Methods : Were there any practical difficulties that could have compromised the study or its implementation? Were these considered in the protocol? Were there any missing values and, if so, was the number of missing values too large to permit meaningful analysis? Was the number of samples (cases or participants) too small to establish meaningful significance? Do the study methods address key potential sources of bias? Were suitable ‘controls’ included in the study? If controls are missing or not appropriate to the study design, we cannot be confident that the results really show what is happening in an experiment. Were the statistical analyses appropriate and applied correctly? Do the authors point out the limitations of methods or tests used? Were the methods referenced and described in sufficient detail for others to repeat or extend the study?

Take a look at the Results section and relevant tables and figures : Is there a clear statement of findings? Were the results expected? Do they make sense? What data supports them? Do the tables and figures clearly describe the data (highlighting trends etc.)? Try to distinguish between what the data show and what the authors say they show (i.e. their interpretation).

Moving on to look in greater depth at the Discussion and Conclusion : Are the results discussed in relation to similar (previous) studies? Do the authors indulge in excessive speculation? Are limitations of the study adequately addressed? Were the objectives of the study met and the hypothesis supported or refuted (and is a clear explanation provided)? Does the data support the authors’ conclusions? Maybe there is only one experiment to support a point. More often, several different experiments or approaches combine to support a particular conclusion. A rule of thumb here is that if multiple approaches and multiple lines of evidence from different directions are presented, and all point to the same conclusion, then the conclusions are more credible. But do question all assumptions. Identify any implicit or hidden assumptions that the authors may have used when interpreting their data. Be wary of data that is mixed up with interpretation and speculation! Remember, just because it is published, does not mean that it is right.

O ther points you should consider when evaluating a research paper : Are there any financial, ethical or other conflicts of interest associated with the study, its authors and sponsors? Are there ethical concerns with the study itself? Looking at the references, consider if the authors have preferentially cited their own previous publications (i.e. needlessly), and whether the list of references are recent (ensuring that the analysis is up-to-date). Finally, from a practical perspective, you should move beyond the text of a research paper, talk to your peers about it, consult available commentaries, online links to references and other external sources to help clarify any aspects you don’t understand.

The above can be taken as a general guide to help you begin to critically evaluate a scientific research paper, but only in the broadest sense. Do bear in mind that the way that research evidence is critiqued will also differ slightly according to the type of study being appraised, whether observational or experimental, and each study will have additional aspects that would need to be evaluated separately. For criteria recommended for the evaluation of qualitative research papers, see the article by Mildred Blaxter (1996), available online. Details are in the References.

Activity 1 Critical appraisal of a scientific research paper

A critical appraisal checklist, which you can download via the link below, can act as a useful tool to help you to interrogate research papers. The checklist is divided into four sections, broadly covering:

• some general aspects
• research design and methodology
• the results
• discussion, conclusion and references.

Science perspective – critical appraisal checklist [ Tip: hold Ctrl and click a link to open it in a new tab. ( Hide tip ) ]

• Identify and obtain a research article based on a topic of your own choosing, using a search engine such as Google Scholar or PubMed (for example).
• The selection criteria for your target paper are as follows: the article must be an open access primary research paper (not a review) containing empirical data, published in the last 2–3 years, and preferably no more than 5–6 pages in length.
• Critically evaluate the research paper using the checklist provided, making notes on the key points and your overall impression.

Critical appraisal checklists are useful tools to help assess the quality of a study. Assessment of various factors, including the importance of the research question, the design and methodology of a study, the validity of the results and their usefulness (application or relevance), the legitimacy of the conclusions, and any potential conflicts of interest, are an important part of the critical appraisal process. Limitations and further improvements can then be considered.

How to critically appraise a qualitative health research study

• PROFESSIONI INFERMIERISTICHE 72(4):283-293
• 72(4):283-293

• Università degli Studi di Milano-Bicocca

• York University

• McMaster University

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations
• Mónica-Paola Quemba-Mesa
• Martha-Inés Bernal-García
• Sara Silva-Ortiz
• Andrea-Lucia Bravo-Sánchez
• Keith K Gentry
• Karen Snyder
• Jennifer Jordan Utley
• Keith Gentry
• Tammy J. LeSage
• Devin Hebble
• Ashlyn Watts
• BMC Palliat Care

• Harvey Max Chochinov

• J HUM NUTR DIET

• Allison Cammer
• Lesley L. Moisey
• Susan M. Jack

• Barbara De Vito

• Emanuela Garrone

• Elizabeth Orr

• Holly Tschirhart
• BMC MED RES METHODOL

• Simon Lewin

• QUAL HEALTH RES

• Int J Evid Base Healthc

• Saverio Tonet
• Ines Bernard

• J CLIN EPIDEMIOL

• Kate Flemming

• ADV NURS SCI
• Margarete Sandelowski
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

• Open access
• Published: 02 September 2024

Using Normalisation Process Theory to explore the contribution of stakeholder workshops to the development and refinement of a complex behavioural intervention: the STAMINA lifestyle intervention

• Sophie Reale   ORCID: orcid.org/0000-0003-2421-7661 1 ,
• Rebecca R. Turner 2 ,
• Liz Steed 3 ,
• Steph J. C. Taylor 3 ,
• Derek J. Rosario 4 ,
• Liam Bourke 1 ,
• Dylan Morrissey 5 , 6 ,
• Aidan Q. Innes 7 &
• Eileen Sutton 8  

Implementation Science Communications volume  5 , Article number:  94 ( 2024 ) Cite this article

5 Altmetric

Metrics details

The National Institute for Health and Care Excellence (NICE) recommend that men with prostate cancer on androgen deprivation therapy (ADT) are offered twice weekly supervised aerobic and resistance exercise to address side effects of treatment. However, supervised exercise is not routinely offered in standard clinical practice. The STAMINA programme grant for applied research (PGfAR) has been designed to evaluate whether this recommendation can be delivered within standard NHS care. This paper describes how future implementation of NICE recommendations within the NHS was explored during complex intervention development to enable evaluation of a lifestyle intervention.

Two stakeholder workshops were conducted to explore factors pertinent to future implementation of the STAMINA Lifestyle intervention (SLI). Normalisation Process Theory (NPT) provided the theoretical framework for discussion and analysis. Stakeholder workshop 1 focussed on intervention coherence and buy-in. Stakeholder workshop 2 explored barriers and facilitators for embedding SLI into the context of the NHS, with delivery partner Nuffield Health, in the future.

Workshops were attended by healthcare professionals ( n  = 16), exercise professionals ( n  = 17), members of public involved in PPI including patients ( n  = 12), health psychologists ( n  = 2), clinical commissioners ( n  = 4), cancer charities ( n  = 3), a cancer alliance ( n  = 1) and health economist ( n  = 1). Stakeholders agreed that professional training packages should emphasise the uniqueness of the SLI and underpinning theory and evidence ( Coherence ). To further engagement, the use of STAMINA champions and information about the delivery partner were recommended to enhance confidence and knowledge ( Cognitive participation ). Furthermore, a simple communication ( Collective Action ) and progress reporting system ( Reflexive Monitoring ) was suggested to fit into existing infrastructure within the NHS and community partner.

Conclusions

Application of NPT within two stakeholder workshops enhanced complex intervention development. Context-specific strategies to support implementation of SLI within the context of a trial were proposed, sensed-checked, and considered acceptable. The organisational implications of embedding and sustaining the intervention in preparation for wider NHS roll-out were considered (if proven to be effective) and will be explored in the qualitative component of a process evaluation underpinned by NPT.

Trial registration

(ISRCTN: 46385239 ). Registered on July 30, 2020.

Peer Review reports

Contributions to the literature

It takes on average 17 years for 14% of original research to be integrated into clinical practice.

We demonstrate how implementation science and behavioural science theories and frameworks can be applied in parallel to optimise intervention development that might contribute to future implementation of complex interventions into the NHS and charity sector (i.e., Nuffield Health), in a time and resource efficient way.

These findings address recognised gaps between research, policy, and practice by identifying acceptable context specific implementation facilitators to support the embedding of supervised exercise into standard NHS prostate cancer care (if found to be clinically and cost effective).

Supervised exercise is a safe and clinically efficacious solution to address the adverse effects of androgen deprivation therapy (ADT) for prostate cancer [ 1 ]. Clinically meaningful improvements in fatigue and cancer specific quality of life are demonstrated following twelve weeks of supervised exercise and dietary support, with improvements being sustained up to 6-month follow-up [ 1 , 2 ]. As such the National Institute for Health and Care Excellence (NICE; NG131 1.4.19) recommend that men with locally advanced and advanced prostate cancer on ADT should be offered supervised, aerobic and resistance exercise at least twice a week for 12 weeks [ 3 ]. However, very few National Health Service (NHS) trusts have established adequate provision for such supervised services according to this recommendation [ 4 ]. This deficit is emblematic of the barriers to translating and embedding high-quality evidence and national recommendations into complex healthcare systems.

At present, it is estimated that less than half of all novel clinical innovations with proven efficacy are translated into routine healthcare [ 5 ]. Furthermore, it takes a considerable length of time for research findings to influence policy and practice [ 6 ]. Fortunately, significant recent advances in Implementation Science have yielded theories and frameworks that guide researchers towards embedding and evaluating interventions in real-world settings [ 7 ]. As such, it is recommended that factors pertinent to the immediate and future implementation of interventions (i.e., within a trial and healthcare setting respectively) are explored early, during complex intervention development. Failure to consider implementation during intervention development and subsequent trial evaluation may slow or halt the uptake and translation of proven healthcare interventions into future routine care.

Normalisation Process Theory (NPT) [ 8 ] was developed to understand and overcome difficulties encountered when implementing complex behavioural interventions into healthcare practice. Recent guidance on intervention development and evaluation stress the importance of encompassing strong theoretical foundations and engaging diverse stakeholders at all stages [ 9 , 10 ]. NPT is a mid-range theory that can be applied at different stages of research to provide an explanatory framework to help understand how social practices are or may be implemented, embedded, and integrated into routine practice [ 11 ]. This contrasts to other frameworks such as the Theoretical Domains Framework (TDF) which take a more (though not solely) individual perspective on behaviour change and were used in this programme of work during earlier parts of the intervention development [ 12 , 13 ]. NPT suggests the implementation of an intervention is operationalised through four main constructs: understanding the way people make sense of the work of implementing and integrating an intervention individually and as a team (construct 1: coherence); how they engage with that intervention (construct 2: cognitive participation); how they enact it in day-to-day practice (construct 3: collective action); and how they appraise its effects (construct 4: reflexive monitoring).

In this paper, we discuss the application of NPT during complex intervention development to identify barriers and facilitators to implementation of the STAMINA lifestyle intervention (Additional File 1) in the NHS.

Study context, design and approvals

This study contributes to a National Institute for Health and Care Research (NIHR) funded Programme Grant for Applied Research (PGfAR) aiming to evaluate the clinical and cost-effectiveness of embedding a tailored lifestyle intervention for men with prostate cancer on ADT into NHS prostate cancer care with delivery partner Nuffield Health (ISRCTN: 46385239) (See Fig.  1 ). The STAMINA lifestyle intervention is a behaviourally informed supervised exercise programme with dietary information endorsed by NHS healthcare professionals and delivered by upskilled community-based personal trainers. The STAMINA complex intervention was developed and refined in accordance with the Medical Research Council (MRC) guidance for the development of complex interventions [ 14 ]. To date, we have published work on the early phases of intervention development which were exploratory in nature and led to the definition of target behaviours and initial drafts of intervention materials [ 12 , 13 ]. In this paper we provide a detailed account of two stakeholder workshops underpinned by NPT to explore future implementation during the later stages of complex intervention development.

Overview of the STAMINA programme grant for applied research

Stakeholders can help to identify priorities, understand contextual problems and help find solutions that may make a difference to future implementation of interventions in the real world [ 14 ]. There are multiple methods for capturing stakeholder’s expert opinion, e.g., interviews, questionnaires, workshops [ 15 ]. Workshops provide a time-efficient method for collecting rich qualitative data from stakeholder groups and provide an opportunity to explore similarities and differences in opinion between and within professional and patient groups. Moreover, workshops enable stakeholders to be part of the decision-making during intervention development which is important for facilitating engagement and future implementation of an intervention or health problem [ 14 ]. This methodology has been used in previous studies developing complex interventions in the context of diabetes [ 16 ] and digital health [ 17 ].

Stakeholder workshop 1 (SW1) was held in February 2019 building on the previously published review of evidence and qualitative exploration of barriers and facilitators to the patient and professional target behaviours (Table  1 ). Stakeholder workshop 2 (SW2) was held in January 2020 ahead of the complex intervention being finalised and evaluated in a pragmatic randomised controlled trial (RCT) (See Fig.  2 ).

Contribution of two stakeholder workshops to the development and refinement of the STAMINA complex intervention

Regulatory and ethical approvals were sought prior to the commencement of research activities from Sheffield Hallam University (Reference: ER10748795) and the NHS (REC reference: 18/NW/0738 / IRAS project ID: 254343). Written informed consent was collected from all participants prior to research activity.

Participants

Each workshop brought together representatives from key stakeholder groups purposively sampled to include NHS healthcare professionals (HCPs) working in prostate cancer care, community-based exercise professionals (EPs), people with advanced prostate cancer (patients) and their families/ carers, behaviour change experts and researchers. Stakeholders were invited by email or face-to-face to a 1-day workshop. A different group of stakeholders were invited to each workshop to capture a large range of knowledge, experience and opinion and to prevent possible time burden associated with attendance. Some of the participants were known to the research team due to their involvement with STAMINA in earlier work packages.

Two semi-structured topic guides underpinned by NPT [ 8 ] were developed to guide stakeholder workshop 1 and 2 (Additional file 2). An additional question related to the content, format and structure of intervention materials was added to support ongoing parallel intervention development.

Workshop manuals were designed to facilitate networking and discussion during the stakeholder workshops. The manuals contained: a summary of the STAMINA programme, speaker biographies, list of attendees, a copy of the slide deck presented including example training and intervention materials, space to record notes and references for further reading. At the end of the workshop participants were provided with a feedback form to capture further feedback, related to the STAMINA Lifestyle Intervention, the STAMINA programme and the delivery of the workshop.

Participants were not provided with any materials before or after the workshop.

Stakeholder workshops were delivered in a conference room in Sheffield. Workshop participants were allocated a group table comprised of representatives from all stakeholder groups for facilitated cross-group discussion [ 18 ] considering impacts across these groups. The format of both workshops included an introduction to the table facilitator and full disclosure of the workshop aims followed by short explanatory presentations on various intervention elements by STAMINA research team members and task-based table discussions led by a facilitator (SR, RRT, ES, LS, ST, LB) and broad topic guide (Additional File 2) [ 19 ]. Each table discussion was followed by a facilitated (DM) whole group discussion as an opportunity for debate, sense checking and conclusion generation. Table and group facilitators were members of the research team including both male and female Research Assistants, Research Fellow, Senior Lecturer, Qualitative Expert, and Professors with the following credentials (MSc, PhD, MD). All facilitators had a minimum 1 year of facilitation experience and were guided by a qualitative research expert (ES). There were no other people in attendance of the workshop, and nobody withdrew from the study.

Notes from the table and whole group discussions were written individually by table facilitators in workbooks and on flip-chart paper (i.e., discussions were not audio recorded or transcribed verbatim). Stakeholders were provided with feedback forms to complete at the end of the session. Moreover, written feedback was collated by table facilitators and the research team to produce a summary that was sent to all stakeholders who were then invited to send any further comments to the team via email.

An inductive and deductive approach to thematic analysis was guided by Braun and Clark’s six phases to thematic analysis [ 20 ]. Researchers familiarised themselves with the data whilst transcribing feedback during the workshop and re-reading whilst collating data to be sent to stakeholders for further comment. NPT constructs and sub-constructs provided a framework for analysis of the stakeholder workshop discussions and feedback forms with coding identifying what type of stakeholder provided the information and on which component of the intervention they were commenting on allowing us to pay attention to uncertainties across the different stakeholder groups and specific elements of the intervention [ 21 ]. Where data was not deemed relevant to the NPT constructs/sub-constructs then this was coded inductively by identifying patterns of potential interest. Codes were then collated with related codes to form groups ( i.e., new themes) which were discussed, refined and named where consensus was achieved between researchers during an iterative process of analysis (Table  2 ). Data was managed using Microsoft Office.

The analysis was led by a female Research Fellow (PhD) with expertise in qualitative research and application of the NPT (ES). Data was cross-checked by two independent female researchers, one a Research Fellow (SR) the other a Research Assistant (RRT) with a minimum of three years’ experience analysing qualitative data and the following credentials, BSc, MSc, PhD. Inconsistencies were resolved through discussion.

Stakeholder Workshop 1

Twenty-eight stakeholders attended SW1 (Table  3 ). Workshop participants discussed the content of the intervention and associated training packages, with a view to its potential integration into routine cancer care. We explored stakeholders’ perceptions of STAMINA across the four NPT constructs and sub-constructs to help understand perceived potential benefits of STAMINA for men with prostate cancer, in addition to factors impacting upon its future implementation (Additional File 3).

Coherence: understanding the purpose, value, and benefits of the STAMINA programme

Within NPT the embedding of a practice (normalization) is made possible by “a set of ideas about its meaning, uses, and utility; and by socially defined and organized competencies” [ 8 ] (p542) which hold the practice together. Stakeholders in workshop one recognised the importance of highlighting STAMINA’s unique selling points in professional training and patient-facing materials—including how the intervention is tailored to address individual needs (differentiation). Further key factors were ensuring that all stakeholders were made aware of the evidence base in support of the intervention (communal specification); that HCPs and EPs understood their individual roles and responsibilities within STAMINA (individual specification); and that patient-facing materials (such as STAMINA information booklets) should highlight the supportive evidence base, illustrated by patient stories, pictures and quotes. Together these factors were seen to make the practice “meaningful”, for example increasing HCP’s confidence in referring patients to the STAMINA Lifestyle Intervention (Additional file 3).

Cognitive participation: getting buy-in, STAMINA champions

This construct explores the “symbolic and real enrolments and engagements of human actors” [ 8 ] (p 243) that enable them to work together to build and sustain a community of practice. It was important to explore participants opinions on the possibility for the introduction of STAMINA in routine prostate cancer care – would HCPs and EPs be prepared to invest time and resource into getting STAMINA up and running [ 11 ], and would men with prostate cancer on ADT be keen to participate? To facilitate HCP and EP buy-in, stakeholders highlighted the importance of enrolling STAMINA “champions” located in both the NHS and Nuffield Health to engender confidence in the intervention (enrolment) and that training would need to be flexible with advanced notice to fit in with existing roles and commitments (initiation, enrolment). It was also highlighted that demonstrating the expertise of Nuffield Health in delivering exercise programmes to clinical populations (e.g., cystic fibrosis, joint pain) and their charitable status would be key to gaining professional trust in the intervention (legitimation). For patients, clear information about the benefits of exercise whilst undergoing ADT were emphasised as being important for patient buy-in with information presented by professionals and in booklet format (enrolment) (Additional file 3).

Collective Action: how STAMINA works in day-to-day practice (including roles/resources), communication pathways

This construct considers how actors work collectively towards specific goals as part of a particular practice [ 8 ]. Discussions focussed largely on early stages of implementation, for example, enabling HCPs and EPs to complete related training packages by offering different modes of deliveries (e.g., face-to-face versus remote) and hosting them on internal systems (interactional workability, contextual integration). Participants in workshop one also identified a need for an established referral and communication pathway, particularly in relation to assessing patient eligibility for participation and clear processes for raising concerns (relational integration; contextual integration). Furthermore, there were discussions about the current and required skillset of different roles within the fitness and wellbeing industry (e.g., physiologists, physiotherapists, and personal trainers) to identify the most suitable role and subsequent training needs to deliver the STAMINA programme – with a particular focus on having the skills and confidence to tailor exercise for men with prostate cancer who often have a number of health comorbidities (skill set workability).

Similarly, from the patient perspective, it was considered important to have a trained exercise professional to deliver the intervention and access to a medical professional for clinical support/ query resolution where required (skill set workability). Moreover, patients believed that the programme would work best if a variety of one-to-one and group sessions were offered (Contextual integration) (Additional file 3).

Reflexive monitoring: processes for measuring outcomes (team and individual) and feeding back to those delivering (HCPs, EPs) or taking part (patients) in STAMINA

The final NPT core construct considers how actors work together to continuously evaluate the outcomes of a practice, and in turn how these impact on its potential for normalization [ 8 ]. Discussions briefly touched on developing clear systems for clinical escalations in emergency scenarios (systemization); as well as methods of assessing the professional training packages (communal appraisal, individual appraisal); and adapting training materials and intervention documents (i.e., progress reports) to fit with existing technological systems (reconfiguration). Factors related to this construct were discussed more extensively in SW2 (Additional file 3).

Intervention refinement following Stakeholder Workshop 1

Following feedback from stakeholders the STAMINA complex intervention was refined to optimise the process of embedding supervised exercise into the prostate cancer care pathway ahead of a pragmatic RCT [ 8 ]. Key changes included hosting high level training on Nuffield Health internal systems and mandating completion by all front of house staff, management, and sales teams to enhance awareness related to operational procedures. Moreover, the decision was made to upskill personal trainers to deliver and support patient exercise behaviour as their role within Nuffield Health more closely aligns with the requirements of delivering SLI compared to that of a physiotherapist or physiologist. Information about Nuffield Health (e.g., charitable status, previous involvement in clinical research and exercise delivery, skillset, and mandatory training) was also added to the HCP training package and patient-facing materials alongside case study examples (quotes, patient stories, pictures etc.) to enhance understanding of the intervention purpose and subsequent increase buy-in.

Stakeholder Workshop 2

Twenty-eight different stakeholders attended SW2 i.e., they had not participated in SW1 (Table  3 ). We built on the feedback from SW1 to frame discussions on the refined intervention. Discussions focussed on further refinement; progress reporting; communication pathways; and involved detailed consideration of the integration of STAMINA in day-to-day practice in both the NHS and charitable sector. There were some brief discussions related to the NPT construct of Coherence, where participants in workshop two re-emphasised the importance of STAMINA’s grounding in a supportive evidence-base and tailoring to fit individual patient’s needs to facilitate normalization (differentiation). Factors related to the construct of Cognitive Participation involved consideration of how the format of the patient facing materials might be refined to enhance acceptability of their use in the gym and prevent unwanted attention from the general public. It was also agreed that information about suitable clothing should be added so that men do not feel the need to invest in expensive kit (enrolment, legitimation) (Additional file 3).

To implement STAMINA, key stakeholders need to work together to achieve common goals. Factors related to the construct Collective action were therefore discussed, including consideration of how HCPs and EPs might work together in the future to communicate key outcomes e.g., sharing patient progress with primary care staff (interactional workability). Specific roles and responsibilities in operationalizing the intervention were explored in detail, with a particular emphasis on streamlining progress reporting systems. Stakeholders also emphasised the role of the patient in taking ownership for recording their progress (skill set workability). The sub-construct of contextual integration focuses on how the introduction of a new practice affects “the mechanisms that link work to existing structures and procedures, and for realizing material and symbolic resources for them” [ 8 ] (p545). HCP and EP attendees highlighted the importance of introducing communication pathways and reporting systems that could integrate within existing systems and that they could easily incorporate within their day-to-day practice, and that these should be clearly set out in initial training packages (contextual integration). This included systems for reporting safety/health issues such as Nuffield Health’s internal Clinical Escalation policy and team (Additional file 3).

Reflexive monitoring: processes for measuring outcomes (team and individual) and feeding back to those delivering (HCPs, CEPs) or taking part (patients) in STAMINA

SW2 involved detailed discussion of factors related to the refinement of systems for collecting information to determine the utility/effectiveness of STAMINA (Additional File 3). There was lengthy consideration of the format, function, language/terminology, and purpose of the progress report e.g., that it should not act as safety reporting tool. Similarly, participants in workshop two proposed and agreed that the STAMINA diary should act as a self-monitoring tool to support patient behaviour change and maintenance and should be separate to the logbooks designed for data capture/ research purposes only (systemization). Leading on from this, stakeholders discussed how patients and EPs should use the report as a tool to aid discussions to evaluate progress (communal appraisal); and that patients should be provided with the opportunity to voice opinions on their personal progress e.g., within the STAMINA diary (individual appraisal). The latter factor linked to the NPT sub-construct of legitimation, in giving men confidence to participate. Possible enhancements to the patient facing materials were also discussed e.g., adding patient stories and photographs (reconfiguration).

Intervention refinement following Stakeholder Workshop 2

Following SW2, the patient facing materials were refined to increase acceptability based on stakeholder recommendations. More specifically, i) the STAMINA diary was reduced from A4 to A5 and wire bound to increase usability, ii) all STAMINA logos were removed from the front and back cover and replaced with standard Nuffield Health branding to reduce the likelihood of patients ‘standing out’ and receiving subsequent unwanted special attention and iii) additional information was added to depict appropriate clothing for exercise to enhance inclusivity. Moreover, an email communication pathway, embedded into current internal systems and IT infrastructure was established for reporting/escalating problems and discussing patient progress between Nuffield Health and the NHS. The method was chosen for its simplicity, safety (e.g., end-to-end encryption) and future proofing (e.g., likely to work in the future regardless of changes made to internal IT systems) (Additional file 3).

This paper describes the application of NPT to complex intervention development for management of ADT side-effects in prostate cancer. Behaviour change theories [ 22 , 23 ] guided the early intervention development stages which were later complemented by Implementation Science, i.e., NPT [ 8 ]. Feedback was captured from 56 stakeholders during two stakeholder workshops to enhance possibilities for normalization, as they “play an important part in feeding back into notions of the coherence and meaningfulness of a practice” [ 8 ]. Discussions and analysis centred on NPT led to further development and refinement of the STAMINA complex lifestyle intervention including patient facing materials and professional training packages. Moreover, consideration for individual roles and infrastructure within the specified contexts led to the identification of a clear and simple communication pathway to fit within the organisational structures of the NHS and delivery partner, Nuffield Health.

At present, the provision of supervised exercise into the prostate cancer care pathway, as recommended by NICE, is rarely offered due to a multitude of barriers related to implementation [ 4 ]. As described previously [ 24 ], NPT was used to identify barriers and facilitators related to practicalities of NHS delivery in a sequential fashion across the two workshops. In the early phases of intervention development (i.e., SW1), the constructs of Coherence and Cognitive Participation (and related sub-constructs) were of particular relevance in our analysis. All stakeholders identified ‘buy-in’ and a good understanding of the underpinning evidence in support of the intervention as a necessary precursor to its implementation. In particular, the introduction of STAMINA Champions, information about the charity sector and clarity on professional roles were collectively identified as ways to increase HCP confidence to recommend and refer patients for exercise. Over the past decade, the use of champions has become a focal point of implementation research with systematic reviews evidencing champions as one of the core determinants of implementation success in healthcare settings [ 25 ]. However, less is known about the use of champions in other contexts. In the present study, EPs agreed champions could be beneficial in the charity sector though greater emphasis was placed on which professionals should be trained to deliver the intervention and where the training was hosted as a facilitator to enhance early ‘buy-in’. These differences highlight the importance of learning about the target contexts so that implementation strategies can be tailored accordingly (i.e., it is not a one size fits all).

The NPT constructs Collective Action and Reflexive Monitoring (plus sub-constructs) were the main discussion points of SW2. Discussions were centred on how to operationalise SLI into routine practice with consideration for future refinements ahead of delivery in the ever-changing context of the NHS and charity sector (Reconfiguration). For example, a simple and clear communication and reporting pathway between HCPs, EPs, and patients was favoured against a more complex reporting system that may require specialist software/ resource and may not be accessible to all contexts involved, now or in the future. Moreover, stakeholders agreed that intervention and research related information should be presented in separate materials to increase internal validity and so that minimal, if any, changes are required to the intervention materials ahead of national scale implementation (if found to be effective) – this is advantageous from both a resource and time perspective. Failure to address context specific changes could constitute a major barrier to appraising the transferability and applicability of findings following our planned RCT and may contribute to furthering the gap between research, policy, and practice [ 26 ]. Instead, the workshops developed our understanding of the local contexts so that a modifiable intervention could be developed and evaluated in our process evaluation to enable future implementation in a timely fashion (if found to be effective).

Strengths & Limitations

Consideration of factors related to implementation as part of an iterative intervention development process was deemed essential for intervention optimisation ahead of testing in the context of a RCT. This blended approach is strengthened by its methodological rigour, time efficiency and likeliness of success compared to more traditional and sequential methods which tend to retrospectively explore implementation following intervention development and efficacy trials. Subsequently, the STAMINA Lifestyle Intervention was found to be feasible and acceptable, and delivered with moderate to high fidelity, when embedded into the NHS and Nuffield Health during a feasibility study [ 27 ].

More specifically, delivering two stakeholder workshops at different phases of intervention development enabled the research team to employ the four constructs of NPT to sequentially explore and identify determinants of (potential) future implementation. Furthermore, offering two workshops enhanced stakeholder reach thus providing valuable insight into patient perspective and organisational specific information which has been found to increase the adoption, implementation, and sustainability of interventions [ 28 ]. Subsequently, attendance at the workshops may have had additional implementation benefits beyond the primary aim focussing on the intervention. For example, workshop attendance may have acted to enhance Coherence and Cognitive Participation and support delivery of the trial as most clinical stakeholders submitted an expression of interest for their NHS Trust to be involved in embedding supervised exercise into the NHS as part of the STAMINA RCT.

Application of the NPT during intervention development enabled us to identify and characterise key barriers and facilitators of future implementation of the STAMINA Lifestyle Intervention and aid intervention development as part of an iterative process. However, in certain instances, application of the NPT was challenging as during the analysis phase some stakeholder feedback was mapped onto multiple constructs/sub-constructs demonstrating the multifaceted interactions of different elements of a complex intervention. For example, discussions around the format and purpose of the patient diary spanned the sub-constructs of legitimation and individual appraisal. Furthermore, the researchers who have a background in behaviour change had a natural desire to code discussion items at the individual level rather than more social processes. This may reflect the researcher’s familiarity with behaviour change framework s and/ or highlight that the TDF is another tool that can be used to assess implementation [ 22 ]. In this study NPT and TDF were used as complimentary to one another during intervention development and have the potential to improve how the interventions was designed through a robust, systematic method underpinned by multi-level theory as has been recommended by others [ 29 ]. We took an inductive and deductive approach to analysis and so both intervention and implementation barriers were explored in parallel to increase the usefulness and policy relevance of the research [ 30 ].

If the STAMINA Lifestyle Intervention is proven to be clinically and cost effective, future implementation work using NPT will be guided by the newly developed and published NPT coding manual – a tool created to provide a consistent and simple set of definitions of the core constructs of the theory to support researchers with qualitative analysis [ 31 ].

In this paper, we discuss how the NPT was applied during complex intervention development and consider possible future implementation of embedding supervised exercise into routine clinical care. This blended approach demonstrates methodological rigour, time efficiency and increased likeliness of success compared to more traditional and sequential methods which tend to retrospectively explore implementation following intervention development and efficacy trials. All four constructs of the NPT were explored to identify context specific barriers and facilitators to future embedding of new pathways of care into the NHS with delivery partner Nuffield Health. The NPT will provide the explanatory framework to help understand how the STAMINA complex intervention is implemented, embedded, and integrated into routine practice in a planned definitive RCT (Trial registration: ISRCTN: 46,385,239).

Availability of data and materials

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Abbreviations

Exercise Professional

Healthcare Professional

Medical Research Council

National Health Service

National Institute for Health and Care Excellence

National Institute for Health and Care Research

Normalisation Process Theory

Programme Grant for Applied Research

Randomised Controlled Trial

Supported exercise TrAining for Men wIth prostate caNcer on Androgen deprivation therapy

Stakeholder Workshop

Theoretical Domains Framework

Bourke L, Smith D, Steed L, Hooper R, Carter A, Catto J, et al. Exercise for men with prostate cancer: A systematic review and meta-analysis. Eur Urol. 2016;69:693–703. Elsevier B.V.

Article   PubMed   Google Scholar  

Nguyen PL, Alibhai SMH, Basaria S, D’Amico AV, Kantoff PW, Keating NL, et al. Adverse effects of androgen deprivation therapy and strategies to mitigate them. Eur Urol. 2015;67:825–36. Elsevier.

Article   CAS   PubMed   Google Scholar  

Recommendations | Prostate cancer: diagnosis and management | Guidance | NICE [Internet]. 2019 [cited 2020 Apr 28]. Available from: https://www.nice.org.uk/guidance/ng131/chapter/recommendations#people-having-hormone-therapy

Bourke L, Turner R, Greasley R, Sutton E, Steed L, Smith D, et al. A multi-centre investigation of delivering national guidelines on exercise training for men with advanced prostate cancer undergoing androgen deprivation therapy in the UK NHS. Jiang BH, editor. PLoS One. 2018;13(7):e0197606. [cited 2020 Apr 24]. Available from: https://doi.org/10.1371/journal.pone.0197606 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510. [cited 2024 Jan 12]. Available from: /pmc/articles/PMC3241518/.

Article   PubMed   PubMed Central   Google Scholar  

Uzochukwu B, Onwujekwe O, Mbachu C, Okwuosa C, Etiaba E, Nyström ME, et al. The challenge of bridging the gap between researchers and policy makers: Experiences of a Health Policy Research Group in engaging policy makers to support evidence informed policy making in Nigeria. Global Health. 2016;12(1):1–15. [cited 2024 Jan 12]. Available from: https://globalizationandhealth.biomedcentral.com/articles/10.1186/s12992-016-0209-1 .

Google Scholar  

Thomas SA, Stefanovska-Petkovska M, Leeman J, Shelton RC, Chambers DA, Glasgow RE. An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Front Publ Health | [cited 2024 Jan 12]. Available from: www.frontiersin.org . 2020;1:134. Available from: www.frontiersin.org

May C, Finch T. Implementing, Embedding, and Integrating Practices: An Outline of Normalization Process Theory. 2009;43(3):535–54. [cited 2024 Jan 12]. Available from https://doi.org/10.1177/0038038509103208 . https://journals.sagepub.com/doi/10.1177/0038038509103208

Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ Brit Med J Publishing Group. 2008;337:979–83.

Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374:n2061. [cited 2021 Oct 5]. Available from: https://www.bmj.com/content/374/bmj.n2061 .

Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al. Normalisation process theory: A framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8(1):1–11. [cited 2020 Jul 7]. Available from: https://link.springer.com/articles/10.1186/1741-7015-8-63 .

Article   Google Scholar  

Turner RR, Arden MA, Reale S, Sutton E, Taylor SJC, Bourke L, et al. The development of a theory and evidence-based intervention to aid implementation of exercise into the prostate cancer care pathway with a focus on healthcare professional behaviour, the STAMINA trial. BMC Health Serv Res. 2021;21(1):273. [cited 2021 Apr 8]. Available from: https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-021-06266-x .

Reale S, Turner RR, Sutton E, Taylor SJC, Bourke L, Morrissey D, et al. Towards implementing exercise into the prostate cancer care pathway: development of a theory and evidence-based intervention to train community-based exercise professionals to support change in patient exercise behaviour (The STAMINA trial). BMC Health Serv Res. 2021;21(1):264. [cited 2021 Apr 8]. Available from: https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-021-06275-w

O’Cathain A, Croot L, Duncan E, Rousseau N, Sworn K, Turner KM, et al. Guidance on how to develop complex interventions to improve health and healthcare. BMJ Open. 2019;9(8):e029954. [cited 2021 Oct 5]. Available from: https://bmjopen.bmj.com/content/9/8/e029954 .

Boaz A, Hanney S, Borst R, O’Shea A, Kok M. How to engage stakeholders in research: design principles to support improvement. Health Res Policy Syst. 2018;16(1):60. https://doi.org/10.1186/s12961-018-0337-6 . PMID: 29996848; PMCID: PMC6042393.

Lowes L, Robling MR, Bennert K, Crawley C, Hambley H, Hawthorne K et al. Involving lay and professional stakeholders in the development of a research intervention for the DEPICTED Study. Health Expectations. 2010 Sept 23. Available from: Involving lay and professional stakeholders in the development of a research intervention for the DEPICTED Study - Lowes - 2011 - Health Expectations - Wiley Online Library

Lupton D. Digital health now and in the future: Findings from a participatory design stakeholder workshop. Dig Health. 2017;3. https://doi.org/10.1177/2055207617740018

Molster C, Youngs L, Hammond E, Dawkins H. Key outcomes from stakeholder workshops at a symposium to inform the development of an Australian national plan for rare diseases. Orphanet J Rare Dis. 2012;7(1):1–10. [cited 2024 Jan 12]. Available from: https://ojrd.biomedcentral.com/articles/10.1186/1750-1172-7-50 .

Reviewing the Disabled Facilities Grant Programme — University of Bristol. [cited 2024 Mar 26]. Available from: https://ojrd.biomedcentral.com/articles/10.1186/1750-1172-7-50

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. [cited 2018 Jun 25]. Available from: http://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa .

Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):117. [cited 2020 Dec 11]. Available from: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-13-117 .

Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7(1):1–17.

The Behaviour Change Wheel: A Guide to Designing Interventions - Susan Michie, Lou Atkins, Robert West - Google Books. [cited 2020 Apr 22]. Available from: https://books.google.co.uk/books?id=am6ytAEACAAJ&dq=The+behaviour+change+wheel&hl=en&sa=X&ved=0ahUKEwjPq6X71fvoAhVllFwKHR5SCIQQ6AEIJzAA

Alharbi TS, Carlström E, Ekman I, Olsson LE. Implementation of person-centred care: management perspective. J Hosp Adm. 2014;3(3):107.

Miech EJ, Rattray NA, Flanagan ME, Damschroder L, Schmid AA, Damush TM. Inside help: An integrative review of champions in healthcare-related implementation. SAGE Open Med. 2018;6:1–11. [cited 2024 Jan 12]. Available from: https://doi.org/10.1177/2050312118773261 .

Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: The Context and Implementation of Complex Interventions (CICI) framework. Implement Sci. 2017;12(1):1–17. [cited 2024 Jan 12]. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-017-0552-5 .

Reale S, Turner RR, Sutton E, Steed L, Taylor SJC, Morrissey D, et al. Embedding supervised exercise training for men on androgen deprivation therapy into standard prostate cancer care: a feasibility and acceptability study (the STAMINA trial). Scientific Reports. 2021;11(1):1–12. [cited 2024 Jan 12]. Available from: https://www.nature.com/articles/s41598-021-91876-y .

Holcomb J, Ferguson GM, Sun J, Walton GH, Highfield L. Stakeholder Engagement in Adoption, Implementation, and Sustainment of an Evidence-Based Intervention to Increase Mammography Adherence Among Low-Income Women. 1988. [cited 2024 Jan 12]. Available from: https://doi.org/10.1007/s13187-021-01988-2

Steed L, Sheringham J, McClatchey K, Hammersley V, Marsh V, Morgan N, et al. IMP2ART: development of a multi-level programme theory integrating the COM-B model and the iPARIHS framework, to enhance implementation of supported self-management of asthma in primary care. Implement Sci Commun. 2023;4(1):1–13. [cited 2024 Apr 19]. Available from: https://implementationsciencecomms.biomedcentral.com/articles/10.1186/s43058-023-00515-2 .

Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. [cited 2021 Feb 25]. Available from: https://pubmed.ncbi.nlm.nih.gov/22310560/ .

May CR, Albers B, Bracher M, Finch TL, Gilbert A, Girling M, et al. Translational framework for implementation evaluation and research: a normalisation process theory coding manual for qualitative research and instrument development. 2021 [cited 2024 Jan 12]; Available from: https://doi.org/10.1186/s13012-022-01191-x

Download references

Acknowledgements

To participants, the PPI group, Nuffield Health, and STAMINA co-applicants. The STAMINA co-applicants include Tom Baker, Professor Janet Brown, Michelle Collinson, Professor Patrick Doherty, Professor Amanda Farrin, Professor Diana Greenfield*, Suzanne Hartley, Professor Jenny Hewison, Professor Malcom Mason, and Dr David Meads.

This project is funded by the National Institute for Health and Care Research (NIHR) under its Programme Grants for Applied Research (PGfAR) (Grant Reference Number RP-PG-1016–20007). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Author information

Authors and affiliations.

Allied Health Professions, College of Health, Wellbeing & Life Sciences, Sheffield Hallam University, Sheffield, UK

Sophie Reale & Liam Bourke

Division of Medical Education, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK

Rebecca R. Turner

Wolfson Institute of Population Health, Queen Mary University of London, London, UK

Liz Steed & Steph J. C. Taylor

Sheffield Teaching Hospitals NHS Foundation Trust, Department of Urology, Royal Hallamshire Hospital, Sheffield, S10 2JF, UK

Derek J. Rosario

Sport and Exercise Medicine, Queen Mary University of London, London, UK

Dylan Morrissey

Physiotherapy Department, Barts Health NHS Trust, London, UK

Nuffield Health, 2 Ashley Avenue, Epsom, Surrey, UK

Aidan Q. Innes

British Population Health Sciences Institute, University of Bristol, Bristol, UK

Eileen Sutton

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization: LS, SJC; Methodology: SR, RT, ES, DM; Formal Analysis: ES, SR, RT; Writing-Original Draft Preparation: SR, ES; Writing-Review & Editing: SR, ES, RRT, SJC, LS, DR, LB, DM, AI; Funding Acquisition: DJR and LB. All authors read and approved the final manuscript.

* Diana Greenfield is a National Institute for Health Research (NIHR) Senior Nurse Research Leader. The views expressed in this article are those of the author and not necessarily those of the NIHR, or the Department of Health and Social Care.

Corresponding author

Correspondence to Sophie Reale .

Ethics declarations

Ethics approval and consent to participate.

Were sought prior to the commencement of research activities from Sheffield Hallam University (Reference: ER10748795) and the NHS (REC reference: 18/NW/0738 / IRAS project ID: 254343).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

43058_2024_629_moesm1_esm.docx.

Additional file 1. TIDIeR Framework of the STAMINA Lifestyle Intervention. This file presents a description of the STAMINA Lifestyle Intervention in accordance with the TIDIeR framework.

43058_2024_629_MOESM2_ESM.docx

Additional file 2. Topic guide. This file contains the topic guide that guided round table discussions at stakeholder workshop 1 and 2. The topic guide is based on the Normalisation Process Theory.

43058_2024_629_MOESM3_ESM.docx

Additional file 3. Stakeholder discussions mapped onto the NPT. This file contains feedback from stakeholders, in stakeholder workshop 1 and 2, mapped onto the Normalisation Process Theory framework.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Reale, S., Turner, R.R., Steed, L. et al. Using Normalisation Process Theory to explore the contribution of stakeholder workshops to the development and refinement of a complex behavioural intervention: the STAMINA lifestyle intervention. Implement Sci Commun 5 , 94 (2024). https://doi.org/10.1186/s43058-024-00629-1

Download citation

Received : 16 January 2024

Accepted : 18 August 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s43058-024-00629-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Intervention development
• Normalisation process theory
• Behaviour change
• Prostate cancer
• Stakeholders
• Complex intervention
• Process evaluation
• Implementation

Implementation Science Communications

ISSN: 2662-2211

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Putting gpt-4o to the sword: a comprehensive evaluation of language, vision, speech, and multimodal proficiency.

1. Introduction

1.1. research purpose, 1.2. related work, 2. language capacity of gpt-4o, 2.1. performance on exams, 2.1.1. performance on usmle, 2.1.2. performance on cfa, 2.1.3. performance on sat, 2.1.4. performance on mbe, 2.2. reasoning, 2.3. language translation, 2.3.1. data, 2.3.2. evaluation method.

• 1 indicates that the vectors are identical.
• 0 indicates that the vectors are orthogonal (i.e., no similarity).
• −1 indicates that the vectors are opposed.

2.3.3. Results

2.3.4. limitations, 3. vision capacity of gpt-4o, 3.1. image classification: fruits classification, 3.2. image classification: driver drowsiness detection, 3.3. image classification: crop disease classification, 3.4. image classification: glaucoma detection, 3.5. image classification: cancer, tumor, and aneurysm detection, 3.6. image captioning, 4. speech capacity of gpt-4o, 4.1. emotion detection, 4.2. accent detection, 5. multimodal capacity of gpt-4o, 5.1. visual question answering, 5.2. vision-language capabilities, 6. implications, limitations, and future work, 6.1. implications, 6.2. limitations, 6.3. future work, 7. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Gemini Team; Anil, R.; Borgeaud, S.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A.M.; Hauth, A.; Millican, K.; et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv 2024 , arXiv:2312.11805. [ Google Scholar ]
• Korinek, A. Language Models and Cognitive Automation for Economic Research ; National Bureau of Economic Research: Cambridge, MA, USA, 2023. [ Google Scholar ]
• Floridi, L.; Chiriatti, M. GPT-3: Its Nature, Scope, Limits, and Consequences. Minds Mach. 2020 , 30 , 681–694. [ Google Scholar ] [ CrossRef ]
• Dillion, D.; Mondal, D.; Tandon, N.; Gray, K. Large Language Models as Moral Experts? GPT-4o Outperforms Expert Ethicist in Providing Moral Guidance ; OSF: Peoria, IL, USA, 2024. [ Google Scholar ] [ CrossRef ]
• Ray, S. Google CEO Says Gemini AI’s ‘Unacceptable’ Responses Offended Users and Showed Bias. 2024. Available online: https://www.forbes.com/sites/siladityaray/2024/02/28/google-ceo-says-gemini-ais-unacceptable-responses-offended-users-and-showed-bias/?sh=250e1a1b1103 (accessed on 15 June 2024).
• Ongsulee, P. Artificial intelligence, machine learning and deep learning. In Proceedings of the 2017 15th International Conference on ICT and Knowledge Engineering (ICT&KE), Bangkok, Thailand, 22–24 November 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 1–6. [ Google Scholar ]
• Thirunavukarasu, A.J.; Ting, D.S.J.; Elangovan, K.; Gutierrez, L.; Tan, T.F.; Ting, D.S.W. Large language models in medicine. Nat. Med. 2023 , 29 , 1930–1940. [ Google Scholar ] [ CrossRef ]
• Hayawi, K.; Shahriar, S. AI Agents from Copilots to Coworkers: Historical Context, Challenges, Limitations, Implications, and Practical Guidelines. Preprints 2024 . [ Google Scholar ] [ CrossRef ]
• Aher, G.V.; Arriaga, R.I.; Kalai, A.T. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. In Proceedings of the 40th International Conference on Machine Learning, PMLR, Honolulu, HI, USA, 23–29 July 2023; pp. 337–371. Available online: https://proceedings.mlr.press/v202/aher23a.html (accessed on 15 June 2024).
• Mannuru, N.R.; Shahriar, S.; Teel, Z.A.; Wang, T.; Lund, B.D.; Tijani, S.; Pohboon, C.O.; Agbaji, D.; Alhassan, J.; Galley, J.; et al. Artificial intelligence in developing countries: The impact of generative artificial intelligence (AI) technologies for development. Inf. Dev. 2023 , 02666669231200628. [ Google Scholar ] [ CrossRef ]
• Lund, B.D.; Wang, T.; Mannuru, N.R.; Nie, B.; Shimray, S.; Wang, Z. ChatGPT and a new academic reality: Artificial Intelligence-written research papers and the ethics of the large language models in scholarly publishing. J. Assoc. Inf. Sci. Technol. 2023 , 74 , 570–581. [ Google Scholar ] [ CrossRef ]
• Hu, B.; Sheng, Q.; Cao, J.; Shi, Y.; Li, Y.; Wang, D.; Qi, P. Bad actor, good advisor: Exploring the role of large language models in fake news detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada, 26–27 February 2024; Volume 38, pp. 22105–22113. [ Google Scholar ]
• Koubaa, A. GPT-4 vs. GPT-3.5: A Concise Showdown. Preprints 2023 . [ Google Scholar ] [ CrossRef ]
• Coyne, S.; Sakaguchi, K.; Galvan-Sosa, D.; Zock, M.; Inui, K. Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction. arXiv 2023 , arXiv:2303.14342. [ Google Scholar ]
• Salman, S.; Liu, X. Overfitting Mechanism and Avoidance in Deep Neural Networks. arXiv 2019 , arXiv:1901.06566. [ Google Scholar ]
• Shen, X.; Wu, Y.; Backes, M.; Zhang, Y. Voice Jailbreak Attacks Against GPT-4o. arXiv 2024 , arXiv:2405.19103. [ Google Scholar ]
• Ying, Z.; Liu, A.; Liu, X.; Tao, D. Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks. arXiv 2024 , arXiv:2406.06302. [ Google Scholar ]
• Kalyanpur, A.; Saravanakumar, K.; Barres, V.; Chu-Carroll, J.; Melville, D.; Ferrucci, D. LLM-ARC: Enhancing LLMs with an Automated Reasoning Critic. arXiv 2024 , arXiv:2406.17663. [ Google Scholar ]
• Zhang, N.; Sun, Z.; Xie, Y.; Wu, H.; Li, C. The latest version ChatGPT powered by GPT-4o: What will it bring to the medical field? Int. J. Surg. 2024 . [ Google Scholar ] [ CrossRef ]
• Wang, H.; Xu, J.; Xie, S.; Wang, R.; Li, J.; Xie, Z.; Zhang, B.; Xiong, C.; Chen, X. M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models. arXiv 2024 , arXiv:2405.15638. [ Google Scholar ]
• Sonoda, Y.; Kurokawa, R.; Nakamura, Y.; Kanzawa, J.; Kurokawa, M.; Ohizumi, Y.; Gonoi, W.; Abe, O. Diagnostic Performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in “Diagnosis Please” Cases. medRxiv 2024 , 2024.05.26.24307915. [ Google Scholar ] [ CrossRef ]
• Singgalen, Y.A. Analyzing an Interest in GPT 4o through Sentiment Analysis using CRISP-DM. J. Inf. Syst. Inform. 2024 , 6 , 882–898. [ Google Scholar ] [ CrossRef ]
• Pang, S.; Nol, E.; Heng, K. ChatGPT-4o for English language teaching and learning: Features, applications, and future prospects. SSRN Sch. Pap. 2024 , 4837988. [ Google Scholar ] [ CrossRef ]
• Xu, S.; Wang, Y.; Liu, D.; Xu, C. Collage Prompting: Budget-Friendly Visual Recognition with GPT-4V. arXiv 2024 , arXiv:2403.11468. [ Google Scholar ]
• Zhou, Y.; Ong, H.; Kennedy, P.; Wu, C.C.; Kazam, J.; Hentel, K.; Flanders, A.; Shih, G.; Peng, Y.; Moy, L.; et al. Evaluating GPT-4V (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs. Radiology 2024 , 311 , e233270. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Allyn, B. Scarlett Johansson Says She Is “Shocked, Angered” over New ChatGPT Voice. Available online: https://www.npr.org/2024/05/20/1252495087/openai-pulls-ai-voice-that-was-compared-to-scarlett-johansson-in-the-movie-her (accessed on 15 June 2024).
• Li, H.; Ding, W.; Kang, Y.; Liu, T.; Wu, Z.; Liu, Z. CTAL: Pre-training cross-modal transformer for audio-and-language representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 7–11 November 2021; pp. 3966–3977. [ Google Scholar ]
• Federation of State Medical Boards and National Board of Medical Examiners. USMLE Step 1 Content Description and General Information. 2024. Available online: https://www.usmle.org (accessed on 15 June 2024).
• Kung, T.H.; Cheatham, M.; Medenilla, A.; Sillos, C.; Leon, L.D.; Elepaño, C.; Madriaga, M.; Aggabao, R.; Diaz-Candido, G.; Maningo, J.; et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digit. Health 2023 , 2 , e0000198. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gilson, A.; Safranek, C.W.; Huang, T.; Socrates, V.; Chi, L.; Taylor, R.A.; Chartash, D. How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med. Educ. 2023 , 9 , e45312. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Brin, D.; Sorin, V.; Vaid, A.; Soroush, A.; Glicksberg, B.S.; Charney, A.W.; Nadkarni, G.; Klang, E. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci. Rep. 2023 , 13 , 16492. [ Google Scholar ] [ CrossRef ]
• Haleem, A.; Javaid, M.; Qadri, M.A.; Suman, R. Understanding the role of digital technologies in education: A review. Sustain. Oper. Comput. 2022 , 3 , 275–285. [ Google Scholar ] [ CrossRef ]
• Callanan, E.; Mbakwe, A.; Papadimitriou, A.; Pei, Y.; Sibue, M.; Zhu, X.; Ma, Z.; Liu, X.; Shah, S. Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams. arXiv 2023 , arXiv:2310.08678. [ Google Scholar ]
• College Board. The SAT Suite of Assessments. Available online: https://www.collegeboard.org (accessed on 15 June 2024).
• OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F.L.; Almeida, D.; Altenschmidt, J.; Altman, S.; et al. GPT-4 Technical Report. arXiv 2024 , arXiv:2303.08774. [ Google Scholar ]
• National Conference of Bar Examiners. MBE Sample Test Questions. Available online: https://www.ncbex.org (accessed on 15 June 2024).
• Griggs, M. Building a Better Bar Exam. Tex. A&M Law Rev. 2019 , 7 , 1. [ Google Scholar ]
• Katz, D.M.; Bommarito, M.J.; Gao, S.; Arredondo, P. Gpt-4 passes the bar exam. Philos. Trans. R. Soc. A 2024 , 382 , 20230254. [ Google Scholar ] [ CrossRef ]
• Huang, J.; Chang, K.C.-C. Towards Reasoning in Large Language Models: A Survey. In Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023, Toronto, ON, Canada, 9–14 July 2023; Rogers, A., Boyd-Graber, J., Okazaki, N., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2023; pp. 1049–1065. [ Google Scholar ]
• Johnson-Laird, P. Deductive reasoning. WIREs Cogn. Sci. 2010 , 1 , 8–17. [ Google Scholar ] [ CrossRef ]
• Hayes, B.K.; Heit, E.; Swendsen, H. Inductive reasoning. WIREs Cogn. Sci. 2010 , 1 , 278–292. [ Google Scholar ] [ CrossRef ]
• Walton, D. Abductive Reasoning ; University of Alabama Press: Tuscaloosa, AL, USA, 2014. [ Google Scholar ]
• Dalvi, B.; Jansen, P.; Tafjord, O.; Xie, Z.; Smith, H.; Pipatanangkura, L.; Clark, P. Explaining Answers with Entailment Trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, 7–11 November 2021; Moens, M.-F., Huang, X., Specia, L., Yih, S.W., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 7358–7370. [ Google Scholar ] [ CrossRef ]
• Weston, J.; Bordes, A.; Chopra, S.; Rush, A.M.; van Merriënboer, B.; Joulin, A.; Mikolov, T. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. arXiv 2015 , arXiv:1502.05698. [ Google Scholar ]
• Sinha, K.; Sodhani, S.; Dong, J.; Pineau, J.; Hamilton, W.L. CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019; Inui, K., Jiang, J., Ng, V., Wan, X., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 4506–4515. [ Google Scholar ] [ CrossRef ]
• Bhagavatula, C.; Bras, R.L.; Malaviya, C.; Sakaguchi, K.; Holtzman, A.; Rashkin, H.; Downey, D.; Yih, W.; Choi, Y. Abductive Commonsense Reasoning. International Conference on Learning Representations. 2019. Available online: https://openreview.net/forum?id=Byg1v1HKDB (accessed on 15 June 2024).
• López Espejel, J.; Ettifouri, E.H.; Yahaya Alassan, M.S.; Chouham, E.M.; Dahhane, W. GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts. Nat. Lang. Process. J. 2023 , 5 , 100032. [ Google Scholar ] [ CrossRef ]
• Khoshafah, F. ChatGPT for Arabic-English Translation: Evaluating the Accuracy. Res. Sq. 2023 . [ Google Scholar ] [ CrossRef ]
• Tiedemann, J. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, 23–25 May 2012; pp. 2214–2218. [ Google Scholar ]
• Kunchukuttan, A.; Mehta, P.; Bhattacharyya, P. The IIT bombay english-hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 7–12 May 2018. [ Google Scholar ]
• Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2–7 June 2019; Volume 1 (long and short papers), pp. 4171–4186. [ Google Scholar ]
• Reimers, N.; Gurevych, I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019; Inui, K., Jiang, J., Ng, V., Wan, X., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 3982–3992. [ Google Scholar ] [ CrossRef ]
• Cubero, S.; Aleixos, N.; Moltó, E.; Gómez-Sanchis, J.; Blasco, J. Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food Bioprocess Technol. 2011 , 4 , 487–504. [ Google Scholar ] [ CrossRef ]
• Jebraeily, Y.; Sharafi, Y.; Teshnehlab, M. Driver drowsiness detection based on convolutional neural network architecture optimization using genetic algorithm. IEEE Access 2024 , 12 , 45709–45726. [ Google Scholar ] [ CrossRef ]
• Hung, L.P.; Alias, S. Beyond Sentiment Analysis: A Review of Recent Trends in Text Based Sentiment Analysis and Emotion Detection. J. Adv. Comput. Intell. Intell. Inform. 2023 , 27 , 84–95. [ Google Scholar ] [ CrossRef ]
• Shahriar, S. GAN computers generate arts? A survey on visual arts, music, and literary text generation using generative adversarial network. Displays 2022 , 73 , 102237. [ Google Scholar ] [ CrossRef ]
• Graham, C.; Roll, N. Evaluating OpenAI’s Whisper ASR: Performance analysis across diverse accents and speaker traits. JASA Express Lett. 2024 , 4 , 025206. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ahamad, A.; Anand, A.; Bhargava, P. AccentDB: A database of non-native english accents to assist neural speech recognition. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, 11–16 May 2020; European Language Resources Association: Marseille, France, 2020; pp. 5351–5358. Available online: https://www.aclweb.org/anthology/2020.lrec-1.659 (accessed on 15 June 2024).
• Yu, W.; Yang, Z.; Li, L.; Wang, J.; Lin, K.; Liu, Z.; Wang, X.; Wang, L. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv 2023 , arXiv:2308.02490. [ Google Scholar ]
• Zhu, N.; Zhang, N.; Shao, Q.; Cheng, K.; Wu, H. OpenAI’s GPT-4o in surgical oncology: Revolutionary advances in generative artificial intelligence. Eur. J. Cancer 2024 , 206 , 114132. [ Google Scholar ] [ CrossRef ]
• Zhu, H.; Wang, Z.; Shi, Y.; Hua, Y.; Xu, G.; Deng, L. Multimodal Fusion Method Based on Self-Attention Mechanism. Wirel. Commun. Mob. Comput. 2020 , 2020 , 8843186. [ Google Scholar ] [ CrossRef ]
• Zhou, K.; Hassan, F.H.; Hoon, G.K. The State of the Art for Cross-Modal Retrieval: A Survey. IEEE Access 2023 , 11 , 138568–138589. [ Google Scholar ] [ CrossRef ]
• Hayawi, K.; Shahriar, S.; Mathew, S.S. The imitation game: Detecting human and AI-generated texts in the era of ChatGPT and BARD. J. Inf. Sci. 2024 , 01655515241227531. [ Google Scholar ] [ CrossRef ]
• Shahriar, S.; Al Roken, N.; Zualkernan, I. Classification of Arabic poetry emotions using deep learning. Computers 2023 , 12 , 89. [ Google Scholar ] [ CrossRef ]
• Sivarajkumar, S.; Kelley, M.; Samolyk-Mazzanti, A.; Visweswaran, S.; Wang, Y. An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing: Algorithm Development and Validation Study. JMIR Med. Inform. 2024 , 12 , e55318. [ Google Scholar ] [ CrossRef ]
• Shahriar, S.; Al Roken, N. How can generative adversarial networks impact computer generated art? Insights from poetry to melody conversion. Int. J. Inf. Manag. Data Insights 2022 , 2 , 100066. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Shahriar, S.; Lund, B.D.; Mannuru, N.R.; Arshad, M.A.; Hayawi, K.; Bevara, R.V.K.; Mannuru, A.; Batool, L. Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency. Appl. Sci. 2024 , 14 , 7782. https://doi.org/10.3390/app14177782

Shahriar S, Lund BD, Mannuru NR, Arshad MA, Hayawi K, Bevara RVK, Mannuru A, Batool L. Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency. Applied Sciences . 2024; 14(17):7782. https://doi.org/10.3390/app14177782

Shahriar, Sakib, Brady D. Lund, Nishith Reddy Mannuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, and Laiba Batool. 2024. "Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency" Applied Sciences 14, no. 17: 7782. https://doi.org/10.3390/app14177782

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Open access
• Published: 02 September 2024

Clinical supervisor’s experiences of peer group clinical supervision during COVID-19: a mixed methods study

• Owen Doody   ORCID: orcid.org/0000-0002-3708-1647 1 ,
• Kathleen Markey   ORCID: orcid.org/0000-0002-3024-0828 1 ,
• James Turner   ORCID: orcid.org/0000-0002-8360-1420 2 ,
• Claire O. Donnell   ORCID: orcid.org/0000-0003-2386-7048 1 &
• Louise Murphy   ORCID: orcid.org/0000-0003-2381-3963 1  

BMC Nursing volume  23 , Article number:  612 ( 2024 ) Cite this article

Metrics details

Providing positive and supportive environments for nurses and midwives working in ever-changing and complex healthcare services is paramount. Clinical supervision is one approach that nurtures and supports professional guidance, ethical practice, and personal development, which impacts positively on staff morale and standards of care delivery. In the context of this study, peer group clinical supervision provides allocated time to reflect and discuss care provided and facilitated by clinical supervisors who are at the same grade/level as the supervisees.

To explore the clinical supervisor’s experiences of peer group clinical supervision a mixed methods study design was utilised within Irish health services (midwifery, intellectual disability, general, mental health). The Manchester Clinical Supervision Scale was used to survey clinical supervisors ( n  = 36) and semi-structured interviews ( n  = 10) with clinical supervisors were conducted. Survey data were analysed through SPSS and interview data were analysed utilising content analysis. The qualitative and quantitative data’s reporting rigour was guided by the CROSS and SRQR guidelines.

Participants generally had a positive encounter when providing clinical supervision. They highly appreciated the value of clinical supervision and expressed a considerable degree of contentment with the supervision they provided to supervisees. The advantages of peer group clinical supervision encompass aspects related to self (such as confidence, leadership, personal development, and resilience), service and organisation (including a positive working environment, employee retention, and safety), and patient care (involving critical thinking and evaluation, patient safety, adherence to quality standards, and elevated levels of care).

There are many benefits of peer group clinical supervision at an individual, service, organisation, and patient level. Nevertheless, there is a need to address a lack of awareness and misconceptions surrounding clinical supervision to create an environment and culture conducive to realising its full potential. It is crucial that clinical supervision be accessible to nurses and midwives of all grades across all healthcare services, with national planning to address capacity and sustainability.

Peer Review reports

Within a dynamic healthcare system, nurses and midwives face growing demands, underscoring the necessity for ongoing personal and professional development. This is essential to improve the effectiveness and efficiency of care delivery for patients, families, and societies. Despite the increased emphasis on increasing the quality and safety of healthcare services and delivery, there is evidence highlighting declining standards of nursing and midwifery care [ 1 ]. The recent focus on re-affirming and re-committing to core values guiding nursing and midwifery practice is encouraging such as compassion, care and commitment [ 2 ], competence, communication, and courage [ 3 ]. However, imposing value statements in isolation is unlikely to change behaviours and greater consideration needs to be given to ways in which compassion, care, and commitment are nurtured and ultimately applied in daily practice. Furthermore, concerns have been raised about global staff shortages [ 4 ], the evidence suggesting several contributing factors such as poor workforce planning [ 5 ], job dissatisfaction [ 6 ], and healthcare migration [ 7 ]. Without adequate resources and staffing, compromising standards of care and threats to patient safety will be imminent therefore the importance of developing effective strategies for retaining competent registered nurses and midwives is paramount in today’s climate of increased staff shortages [ 4 ]. Clinical supervision serves as a means to facilitate these advancements and has been linked to heightened job satisfaction, enhanced staff retention, improved staff effectiveness, and effective clinical governance, by aiding in quality improvements, risk management, and heightened accountability [ 8 ].

Clinical supervision is a key component of professional practice and while the aim is largely known, there is no universally accepted definition of clinical supervision [ 8 ]. Clinical supervision is a structured process where clinicians are allowed protected time to reflect on their practice within a supportive environment and with the purpose of developing high-quality clinical care [ 9 ]. Recent literature published on clinical supervision [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] highlights the advantages and merits of clinical supervision. However, there are challenges also identified such as a lack of consensus regarding the meaning and goal, implementation issues, variations in approaches in its operationalisation, and an absence of research evidence on its effectiveness. Duration and experience in clinical supervision link to positive benefits [ 8 ], but there is little evidence of how clinical supervision altered individual behaviours and practices. This is reinforced by Kuhne et al., [ 15 ] who emphasise that satisfaction rather than effectiveness is more commonly examined. It is crucial to emphasise that reviews have pinpointed that clinical supervision lowers the risks of adverse patient outcomes [ 9 ] and demonstrates enhancements in the execution of certain care processes. Peer group clinical supervision is a form of clinical supervision whereby two or more practitioners engage in a supervision or consultation process to improve their professional practice [ 17 ]. There is limited evidence regarding peer group clinical supervision and research on the experiences of peer clinical supervision and stakeholders is needed [ 13 ]. In Ireland, peer group clinical supervision has been recommended and guidelines have been developed [ 18 ]. In the Irish context, peer clinical supervision is where both clinical supervisees and clinical supervisors are peers at the same level/grade. However, greater evidence is required to inform future decisions on the implementation of peer group clinical supervision and the purpose of this study is to explore clinical supervisors’ experiences of peer group clinical supervision. As the focus is on peer group supervisors and utilising mixed methods the experiences of the other stakeholders were investigated and reported separately.

A mixed methods approach was used (survey and semi-structured interviews) to capture clinical supervisor’s experiences of clinical supervision. The study adhered to the Consensus-Based Checklist for Reporting of Survey Studies guidelines [ 19 ] (Supplementary File S1 ) and Standards for Reporting Qualitative Research guidelines [ 20 ] (Supplementary File S2 ).

Participants

This study was conducted with participants who successfully completed a professionally credited award: clinical supervision module run by a university in Ireland (74 clinical supervisors across 5 programmes over 3 years). The specific selection criteria for participants were that they were registered nurses/midwives delivering peer group clinical supervision within the West region of Ireland. The specific exclusion criteria were as follows: (1) nurses and midwives who haven’t finished the clinical supervision module at the University, (2) newly appointed peer group clinical supervisors who have yet to establish their groups and initiate the delivery of peer group clinical supervision.

Measures and procedures

The Manchester Clinical Supervision Scale-26 was used to survey participants in February/March 2022 and measure the peer group clinical supervisors’ overall experiences of facilitating peer group clinical supervision. The Manchester Clinical Supervision Scale-26 is a validated 26-item self-report questionnaire with a Likert-type (1–5) scale ranging from strongly disagree (1) to strongly agree (5) [ 21 ]. The Manchester Clinical Supervision Scale-26 measures the efficiency of and satisfaction with supervision, to investigate the skills acquisition aspect of clinical supervision and its effect on the quality of clinical care [ 21 ]. The instrument consists of two main sections to measure three (normative, restorative, and formative) dimensions of clinical supervision utilising six sub-scales: (1) trust and rapport, (2) supervisor advice/support, (3) improved care/skills, (4) importance/value of clinical supervision, (5) finding time, (6) personal issues/reflections and a total score for the Manchester Clinical Supervision Scale-26 is also calculated. Section two consisted of the demographic section of the questionnaire and was tailored to include eight demographic questions concerning the supervisor’s demographics, supervisee characteristics, and characteristics of clinical supervision sessions. There were also two open field questions on the Manchester Clinical Supervision Scale-26 (model of clinical supervision used and any other comments about experience of peer group clinical supervision). The main question about participants’ experiences with peer clinical supervision was “What was your experience of peer clinical supervision?” This was gathered through individual semi-structured interviews lasting between 20 and 45 min, in March/April 2022 (Supplementary file 3 ).

Ethical considerations

Health service institutional review boards of two University hospitals approved this study (Ref: 091/19 and Ref: C.A. 2199). Participants were recruited after receiving a full explanation of the study’s purpose and procedure and all relevant information. Participants were aware of potential risks and benefits and could withdraw from the study, or the survey could be stopped at any time. Informed consent was recorded, and participant identities were protected by using a pseudonym to protect anonymity.

Data analysis method

Survey data was analysed using the data analysis software package Statistical Package for the Social Sciences, version 26 (SPSS Inc., Chicago, Il, USA). Descriptive analysis was undertaken to summarise responses to all items and categorical variables (nominal and ordinal) were analysed using frequencies to detail the number and percentage of responses to each question. Scores on the Manchester Clinical Supervision Scale-26 were reverse scored for 9 items (Q1-Q6, Q8, Q20,21) and total scores for each of the six sub-scales were calculated by adding the scores for each item. Raw scores for the individual sub-scales varied in range from 0 to 20 and these raw scores were then converted to percentages which were used in addition to the raw scores for each sub-scale to describe and summarise the results of the Manchester Clinical Supervision Scale-26. Cronbach’s alpha coefficient was undertaken with the 26 questions included within the Manchester Clinical Supervision Scale-26 and more importantly with each of the dimensions in the Manchester Clinical Supervision Scale-26. The open-ended questions on the Manchester Clinical Supervision Scale-26 and interviews were analysed using content analysis guided by Colorafi and Evans [ 22 ] and categories were generated using their eight steps, (1) creating a coding framework, (2) adding codes and memos, (3) applying the first level of coding, (4) categorising codes and applying the second level of coding, (5) revising and redefining the codes, (6) adding memos, (7) visualising data and (8) representing the data.

Research rigour

To ensure the validity and rigour of this study the researchers utilised the Manchester Clinical Supervision Scale-26 a recognised clinical supervision tool with good reliability and wide usage. Interviews were recorded, transcribed, and verified by four participants, data were collected until no new components appeared, data collection methods and analysis procedures were described, and the authors’ biases were minimised throughout the research process. The Manchester Clinical Supervision Scale-26 instrument internal consistency reliability was assessed which was overall good (α = 0.878) with individual subscale also good e.g., normative domain 0.765, restorative domain 0.864, and formative domain 0.900. Reporting rigour was demonstrated using the Consensus-Based Checklist for Reporting of Survey Studies guidelines [ 19 ] and Standards for Reporting Qualitative Research guidelines [ 20 ].

Quantitative data

Participant and clinical supervision characteristics.

Thirty-six of the fifty-two (69.2%) peer group clinical supervisors working across a particular region of Ireland responded to the Manchester Clinical Supervision Scale-26 survey online via Qualtrics. Table 1 identifies the demographics of the sample who were predominantly female (94.4%) with a mean age of 44.7 years (SD. 7.63).

Peer group clinical supervision session characteristics (Table  2 ) highlight over half of peer group clinical supervisors ( n  = 20, 55.6%) had been delivering peer group clinical supervision for less than one year and were mainly delivered to female supervisees ( n  = 28, 77.8%). Most peer group clinical supervision sessions took place monthly ( n  = 32, 88.9%) for 31–60 min ( n  = 27, 75%).

Manchester Clinical Supervision Scale-26 results

Participants generally viewed peer group clinical supervision as effective (Table  3 ), the total mean Manchester Clinical Supervision Scale-26 score among all peer group clinical supervisors was 76.47 (SD. 12.801) out of 104, Surpassing the clinical supervision threshold score of 73, which was established by the developers of the Manchester Clinical Supervision Scale-26 as the benchmark indicating proficient clinical supervision provision [ 21 ]. Of the three domains; normative, formative, and restorative, the restorative domain scored the highest (mean 28.56, SD. 6.67). The mean scores compare favourably to that of the Manchester Clinical Supervision Scale-26 benchmark data and suggest that the peer group clinical supervisors were satisfied with both the level of support, encouragement, and guidance they provided and the level of trust/rapport they had developed during the peer group clinical supervision sessions. 83.3% ( n  = 30) of peer group clinical supervisors reported being either very satisfied ( n  = 12, 33.3%) or moderately satisfied ( n  = 18, 50%) with the peer group clinical supervision they currently delivered. Within the peer group clinical supervisor’s supervisee related issues ( n  = 17, 47.2%), work environment-related issues ( n  = 16, 44.4%), staff-related issues ( n  = 15, 41.7%) were reported as the most frequent issues, with patient/client related issues being less frequent ( n  = 8, 22.2%). The most identified model used to facilitate peer group clinical supervision was the Proctors model ( n  = 8, 22.22%), which was followed by group ( n  = 2, 5.55%), peer ( n  = 2, 5.55%), and a combination of the seven-eyed model of clinical supervision and Proctors model ( n  = 1, 2.77%) with some not sure what model they used ( n  = 2, 5.553%) and 58.33% ( n  = 21) did not report what model they used.

Survey open-ended question

‘Please enter any additional comments , which are related to your current experience of delivering Peer Group Clinical Supervision.’ There were 22 response comments to this question, which represented 61.1% of the 36 survey respondents, which were analysed using content analysis guided by Colorafi & Evans [ 22 ]. Three categories were generated. These included: personal value/benefit of peer group clinical supervision, challenges with facilitating peer group clinical supervision, and new to peer group clinical supervision.

The first category ‘personal value/benefit of peer group clinical supervision’ highlighted positive experiences of both receiving and providing peer group clinical supervision. Peer group clinical supervisors reported that they enjoyed the sessions and found them both worthwhile and beneficial for both the group and them as peer group clinical supervisors in terms of creating a trusted supportive group environment and motivation to develop. Peer group clinical supervision was highlighted as very important for the peer group clinical supervisors working lives and they hoped that there would be more uptake from all staff. One peer group clinical supervisor expressed that external clinical supervision was a ‘lifeline’ to shaping their supervisory journey to date.

The second category ‘challenges with facilitating peer group clinical supervision’, identified time constraints, lack of buy-in/support from management, staff shortages, lack of commitment by supervisees, and COVID-19 pandemic restrictions and related sick leave, as potential barriers to facilitating peer group clinical supervision. COVID-19 was perceived to have a negative impact on peer group clinical supervision sessions due to staff shortages, which resulted in difficulties for supervisees attending the sessions during work time. Peer group clinical supervisors felt that peer group clinical supervision was not supported by management and there was limited ‘buy-in’ at times. There was also a feeling expressed that peer group clinical supervision was in its infancy, as COVID-19 and its related restrictions impacted on this by either slowing down the process of commencing peer group clinical supervision in certain areas or having to move online. However, more recently improvements in managerial support and supervisee engagement with the peer group clinical supervision process are noted.

The final category ‘new to peer group clinical supervision’ highlighted that some peer group clinical supervisors were new to the process of providing peer group clinical supervision and some felt that this survey was not a true reflection of their experience of delivering peer group clinical supervision, as they were not fully established yet as clinical supervisors due to the impact of COVID-19. Peer group clinical supervisors identified that while they were new to providing peer group clinical supervision, they were enjoying it and that it was a learning curve for them.

Qualitative data

The qualitative phase explored peer group clinical supervisors’ ( n  = 10) own experiences of preparation received and experiences of being a peer group clinical supervisor. Three themes were identified through data analysis, building the foundations, enacting engagement and actions, and realities (Table  4 ).

Building the foundations

This theme highlights the importance of prior knowledge, awareness, and training but also the recruitment process and education in preparing peer group clinical supervisors.

Knowledge and awareness

Participant’s prior knowledge and awareness of peer group clinical supervision was mixed with some reporting having little or no knowledge of clinical supervision.

I’m 20 years plus trained as a nurse , and I had no awareness of clinical supervision beforehand , I really hadn’t got a clue what all of this was about , so it was a very new concept to me (Bernie) .

Others were excited about peer group clinical supervision and while they could see the need they were aware that there may be limited awareness of the value and process of clinical supervision among peers.

I find that there’s great enthusiasm and passion for clinical supervision as it’s a great support mechanism for staff in practice , however , there’s a lack of awareness of clinical supervision (Jane) .

Recruitment

Some participants highlighted that the recruitment process to become a peer group clinical supervisor was vague in some organisations with an unclear and non-transparent process evident where people were chosen by the organisation’s management rather than self-selecting interested parties.

It was just the way the training was put to the people , they were kind of nominated and told they were going and there was a lot of upset over that , so they ended up in some not going at all (Ailbhe) .

In addition, the recruitment process was seen as top loaded where senior grades of staff were chosen, and this limited staff nurse grade opportunities where there was a clear need for peer group clinical supervisors and support.

We haven’t got down to the ground level like you know we’ve done the directors , we’ve done the CNM3s the CNM2s we are at the CNM1s , so we need to get down to the staff nurse level so the nurses at the direct frontline are left out and aren’t receiving supervision because we don’t have them trained (Bernie) .

Training and education

Participants valued the training and education provided but there was a clear sense of ‘imposter syndrome’ for some peer group clinical supervisors starting out. Participants questioned their qualifications, training duration, and confidence to undertake the role of peer group clinical supervisor.

Because it is group supervision and I know that you know they say that we are qualified to do supervision and you know we’re now qualified clinical supervisors but I’m not sure that a three-month module qualifies you to be at the top of your game (Maria) .

Participants when engaged in the peer group clinical supervisor educational programme did find it beneficial and the true benefit was the actual re-engagement in education and published evidence along with the mix of nursing and midwifery practice areas.

I found it very beneficial , I mean I hadn’t been engaged in education here in a while , so it was great to be back in that field and you know with the literature that’s big (Claire) .

Enacting engagement and actions

This theme highlights the importance of forming the groups, getting a clear message out, setting the scene, and grounding the group.

Forming the groups

Recruitment for the group was of key importance to the peer group clinical supervisor and they all sent out a general invitation to form their group. Some supervisors used invitation letters or posters in addition to a general email and this was effective in recruiting supervisees.

You’re reaching out to people , I linked in with the ADoN and I put together a poster and circulated that I wasn’t ‘cherry picking , and I set up a meeting through Webex so people could get a sense of what it was if they were on the fence about it or unsure if it was for them (Karen) .

In forming the peer clinical supervision groups consideration needs to be given to the actual number of supervisees and participants reported four to six supervisees as ideal but that number can alter due to attendance.

The ideal is having five or six consistent people and that they all come on board and that you get the dynamics of the group and everything working (Claire) .

Getting a clear message out

Within the recruitment process, it was evident that there was a limited and often misguided understanding or perception of peer group clinical supervision.

Greater awareness of what actually clinical supervision is , people misjudge it as a supervision where someone is appraising you , when in fact it is more of a support mechanism , I think peer support is the key element that needs to be brought out (Jane) .

Given the lack of clarity and understanding regarding peer group clinical supervision, the participants felt strongly that further clarity is needed and that the focus needs to be on the support it offers to self, practice, and the profession.

Clinical supervision to me is clinical leadership (Jane) .

Setting the scene and grounding the group

In the initial phase of the group coming together the aspect of setting the scene and grounding the group was seen as important. A key aspect of this process was establishing the ground rules which not only set the boundaries and gave structure but also ensured the adoption of principles of trust, confidentiality, and safety.

We start with the ground rules , they give us structure it’s our contract setting out the commitment the expectation for us all , and the confidentiality as that’s so important to the trust and safety and building the relationships (Brid) .

Awareness of group dynamics is important in this process along with awareness of the group members (supervisees) as to their role and expectations.

I reiterate the role of each person in relation to confidentiality and the relationship that they would have with each other within the group and the group is very much aware that it is based on respect for each person’s point of view people may have a fear of contributing to the group and setting the ground rules is important (Jane) .

To ground the group, peer group clinical supervisors saw the importance of being present and allowing oneself to be in the room. This was evident in the time allocated at the start of each session to allow ‘grounding’ to occur in the form of techniques such as a short meditation, relaxation, or deep breathing.

At the start , I do a bit of relaxation and deep breathing , and I saw that with our own external supervisor how she settled us into place so very much about connecting with your body and you’ve arrived , then always come in with the contract in my first sentence , remember today you know we’re in a confidential space , of course , you can take away information , but the only information you will take from today is your own information and then the respect aspect (Mary Rose) .

This settling in and grounding was seen as necessary for people to feel comfortable and engage in the peer group clinical supervision process where they could focus, be open, converse, and be aware of their role and the role of peer group clinical supervision.

People have to be open, open about their practice and be willing to learn and this can only occur by sharing, clinical supervision gives us the space to do it in a space where we know we will be respected, and we can trust (Claire) .

This theme highlights the importance of the peer group clinical supervisors’ past experiences, delivering peer group clinical supervision sessions, responding to COVID-19, personal and professional development, and future opportunities.

Past experiences

Past experiences of peer group clinical supervisors were not always positive and for one participant this related to the lack of ground rules or focus of the sessions and the fact it was facilitated by a non-nurse.

In the past , I suppose I would have found it very frustrating as a participant because I just found that it was going round in circles , people moaning and you know it wasn’t very solution focused so I came from my situation where I was very frustrated with clinical supervision , it was facilitated by somebody that was non-nursing then it wasn’t very , there wasn’t the ground rules , it was very loose (Caroline) .

However, many did not have prior experience of peer group clinical supervision. Nonetheless, through the education and preparation received, there was a sense of commitment to embrace the concept, practice, and philosophy.

I did not really have any exposure or really much information on clinical supervision , but it has opened my eyes , and as one might say I am now a believer (Brid) .

Delivering peer group clinical supervision

In delivering peer group clinical supervision, participants felt supervisees were wary, as they did not know what peer group clinical supervision was, and they had focused more on the word supervision which was misleading to them. Nonetheless, the process was challenging, and buy-in was questioned at an individual and managerial level.

Buy-in wasn’t great I think now of course people will blame the pandemic , but this all happened before the pandemic , there didn’t seem to be you know , the same support from management that I would have expected so I kind of understood it in a way because then there wasn’t the same real respect from the practitioners either (Mary Rose) .

From the peer group clinical supervisor’s perspective, they were all novices in delivering/facilitating peer group clinical supervision sessions, and the support of the external clinical supervisors, and their own peer group clinical supervision sessions were invaluable along with a clinical supervision model.

Having supervision myself was key and something that is vital and needed , we all need to look at our practice and how we work it’s no good just facilitating others without being part of the process yourself but for me I would say the three principles of clinical supervision , you know the normative , formative and restorative , I keep hammering that home and bring that in regularly and revisit the contract and I have to do that often you know (Claire) .

All peer group clinical supervisors commented on the preparation for their peer group clinical supervision sessions and the importance of them having the right frame of mind and that often they needed to read over their course work and published evidence.

I want everybody to have a shared voice and you know that if one person , there is something that somebody feels very strongly and wants to talk about it that they e-mail in advance like we don’t have a set agenda but that’s agreed from the participant at the start (Caroline) .

To assist this, the peer group clinical supervisors noted the importance of their own peer group clinical supervision, the support of their peers, and external clinical supervisors. This preparation in an unpredictable situation can be difficult but drawing on one’s experience and the experience within the group can assist in navigating beyond unexpected situations.

I utilise the models of clinical supervision and this helps guide me , I am more of a facilitator of the group we are experts in our own area and our own role but you can only be an expert if you take the time to examine your practice and how you operate in your role (Brid) .

All clinical supervisors noted that the early sessions can be superficial, and the focus can be on other practice or management issues, but as time moves on and people become more engaged and involved it becomes easier as their understanding of supervision becomes clearer. In addition, there may be hesitancy and people may have difficulty opening up with certain people in the group and this is a reality that can put people off.

Initially there was so much managerial bashing and I think through supervision , I began to kind of think , I need the pillars of supervision , the governance , bringing more knowledge and it shifted everything in the room , trying to marry it with all the tensions that people have (Mary Rose) .

For some clinical supervisors, there were expected and unexpected challenges for them as clinical supervisors in terms of the discussions veering off course and expectations of their own ability.

The other big challenge is when they go off , how do you bring him back , you know when they veer off and you’re expected to be a peer , but you have to try and recoil that you have to get the balance with that right (Mary Rose) .

While peer group clinical supervision is accepted and seen as a valuable process by the peer group clinical supervisors, facilitating peer group supervision with people known to you can be difficult and may affect the process.

I’d love to supervise a group where I actually don’t know the people , I don’t know the dynamics within the group , and I’d love to see what it would be like in a group (Bernie) .

Of concern to clinical supervisors was the aspect of non-attendance and while there may be valid reasons such as COVID-19 the absence of a supervisee for several sessions can affect the group dynamics, especially if the supervisee has only engaged with early group sessions.

One of the ones that couldn’t attend because of COVID and whatever , but she’s coming to the next one and I just feel there’s a lot of issues in her area and I suppose I’m mindful that I don’t want that sort of thing to seep in , so I suppose it’s just for me just to keep reiterating the ground rules and the boundaries , that’s something I just have to manage as a facilitator , but what if they don’t attend how far will the group have progressed before she attends (Caroline) .

Responding to COVID-19

The advent of COVID-19 forced peer group clinical supervisors to find alternative means of providing peer group clinical supervision sessions which saw the move from face-to-face to online sessions. The online transition was seen as seamless for many established groups while others struggled to deliver sessions.

With COVID we did online for us it was fine because we were already formed (Corina) .

While the transition may have been positive many clinical supervisors came across issues because they were using an online format that would not be present in the face-to-face session.

We did have a session where somebody was in the main office and they have a really loud booming voice and they were saying stuff that was not appropriate to say outside of clinical supervision and I was like are you in the office can you lower it down a bit can you put your headphones on (Maria) .

However, two peer group clinical supervisors ceased or hasted the progress of rolling out peer group clinical supervision sessions mainly due to redeployment and staff availability.

With COVID it just had to be canceled here , it’s just the whole thing was canceled so it was very , very difficult for people (Mary Rose) .

It was clear from clinical supervisors that online sessions were appropriate but that they felt they were only appropriate for existing established groups that have had the opportunity to build relationships, develop trust, embed the ground rules, and create the space for open communication and once established a combined approach would be appropriate.

Since we weren’t as established as a group , not everybody knew each other it would be difficult to establish that so we would hold off/reschedule , obviously COVID is a major one but also I suppose if you have an established group now , and again , you could go to a remote one , but I felt like since we weren’t established as a group it would be difficult to develop it in that way (Karen) .

Within practice COVID-19 took priority and other aspects such as peer group clinical supervision moved lower down on the priority list for managers but not for the clinical supervisors even where redeployment occurred.

With COVID all the practical side , if one of the managers is dealing with an outbreak , they won’t be attending clinical supervision , because that has to be prioritised , whereas we’ve prioritised clinical supervision (Maria) .

The valuing of peer group clinical supervision was seen as important by clinical supervisors, and they saw it as particularly needed during COVID-19 as staff were dealing with many personal and professional issues.

During the height of COVID , we had to take a bit of a break for four months as things were so demanding at work for people but then I realised that clinical supervision was needed and started back up and they all wanted to come back (Brid) .

Having peer group clinical supervision during COVID-19 supported staff and enabled the group to form supportive relationships.

COVID has impacted over the last two years in every shape and they needed the supervision and the opportunity to have a safe supportive space and it gelled the group I think as we all were there for each other (Claire) .

While COVID-19 posed many challenges it also afforded clinical supervisors and supervisees the opportunity for change and to consider alternative means of running peer group clinical supervision sessions. This change resulted in online delivery and in reflecting on both forms of delivery (face-to-face and online) clinical supervisors saw the benefit in both. Face-to-face was seen as being needed to form the group and then the group could move online once the group was established with an occasional periodic face-to-face session to maintain motivation commitment and reinforce relationships and support.

Online formats can be effective if the group is already established or the group has gone through the storming and forming phase and the ground rules have been set and trust built , then I don’t see any problem with a blended online version of clinical supervision , and I think it will be effective (Jane) .

Personal and professional development

Growth and development were evident from peer group clinical supervisors’ experiences and this growth and development occurred at a personal, professional, and patient/client level. This development also produced an awakening and valuing of one’s passion for self and their profession.

I suppose clinical supervision is about development I can see a lot of development for me and my supervisees , you know personally and professionally , it’s the support really , clinical supervision can reinvigorate it’s very exciting and a great opportunity for nursing to support each other and in care provision (Claire) .

A key to the peer group clinical supervisor’s development was the aspect of transferable skills and the confidence they gained in fulfilling their role.

All of these skills that you learn are transferable and I am a better manager because of clinical supervision (Maria) .

The confidence and skills gained translated into the clinical supervisor’s own practice as a clinical practitioner and clinical supervisor but they were also realistic in predicting the impact on others.

I have empowered my staff , I empower them to use their voice and I give my supervisees a voice and hope they take that with them (Corina) .

Fundamental to the development process was the impact on care itself and while this cannot always be measured or identified, the clinical supervisors could see that care and support of the individual practitioner (supervisee) translated into better care for the patient/client.

Care is only as good as the person delivering it and what they know , how they function and what energy and passion they have , and clinical supervision gives the person support to begin to understand their practice and how and why they do things in a certain way and when they do that they can begin to question and even change their way of doing something (Brid) .

Future opportunities

Based on the clinical supervisor’s experiences there was a clear need identified regarding valuing and embedded peer group clinical supervision within nursing/midwifery practice.

There has to be an emphasis placed on supervision it needs to be part of the fabric of a service and valued by all in that service , we should be asking why is it not available if it’s not there but there is some work first on promoting it and people knowing what it actually is and address the misconceptions (Claire) .

While such valuing and buy-in are important, it is not to say that all staff need to have peer group clinical supervision so as to allow for personal choice. In addition, to value peer group clinical supervision it needs to be evident across all staffing grades and one could question where the best starting point is.

While we should not mandate that all staff do clinical supervision it should become embedded within practice more and I suppose really to become part of our custom and practice and be across all levels of staff (Brid) .

When peer group clinical supervision is embedded within practice then it should be custom and practice, where it is included in all staff orientations and is nationally driven.

I suppose we need to be driving it forward at the coal face at induction , at orientation and any development for the future will have to be driven by the NMPDUs or nationally (Ailbhe) .

A formalised process needs to address the release of peer group clinical supervisors but also the necessity to consider the number of peer group clinical supervisors at a particular grade.

The issue is release and the timeframe as they have a group but they also have their external supervision so you have to really work out how much time you’re talking about (Maria) .

Vital within the process of peer group clinical supervision is receiving peer group clinical supervision and peer support and this needs to underpin good peer group clinical supervision practice.

Receiving peer group supervision helps me , there are times where I would doubt myself , it’s good to have the other group that I can go to and put it out there to my own group and say , look at this , this is what we did , or this is what came up and this is how (Bernie) .

For future roll out to staff nurse/midwife grade resourcing needs to be considered as peer group clinical supervisors who were managers could see the impact of having several peer group clinical supervisors in their practice area may have on care delivery.

Facilitating groups is an issue and needs to be looked at in terms of the bigger picture because while I might be able to do a second group the question is how I would be supported and released to do so (Maria) .

While there was ambiguity regarding peer group clinical supervision there was an awareness of other disciplines availing of peer group clinical supervision, raising questions about the equality of supports available for all disciplines.

I always heard other disciplines like social workers would always have been very good saying I can’t meet you I have supervision that day and I used to think my God what’s this fabulous hour that these disciplines are getting and as a nursing staff it just wasn’t there and available (Bernie) .

To address this equity issue and the aspect of low numbers of certain grades an interdisciplinary approach within nursing and midwifery could be used or a broader interdisciplinary approach across all healthcare professionals. An interdisciplinary or across-services approach was seen as potentially fruitful.

I think the value of interprofessional or interdisciplinary learning is key it addresses problem-solving from different perspectives that mix within the group is important for cross-fertilisation and embedding the learning and developing the experience for each participant within the group (Jane) .

As we move beyond COVID-19 and into the future there is a need to actively promote peer group clinical supervision and this would clarify what peer group clinical supervision actually is, its uptake and stimulate interest.

I’d say it’s like promoting vaccinations if you could do a roadshow with people , I think that would be very beneficial , and to launch it , like you have a launch an official launch behind it (Mary Rose) .

The advantages of peer group clinical supervision highlighted in this study pertain to self-enhancement (confidence, leadership, personal development, resilience), organisational and service-related aspects (positive work environment, staff retention, safety), and professional patient care (critical thinking and evaluation, patient safety, adherence to quality standards, elevated care standards). These findings align with broader literature that acknowledges various areas, including self-confidence and facilitation [ 23 ], leadership [ 24 ], personal development [ 25 ], resilience [ 26 ], positive/supportive working environment [ 27 ], staff retention [ 28 ], sense of safety [ 29 ], critical thinking and evaluation [ 30 ], patient safety [ 31 ], quality standards [ 32 ] and increased standards of care [ 33 ].

In this study, peer group clinical supervision appeared to contribute to the alleviation of stress and anxiety. Participants recognised the significance of these sessions, where they could openly discuss and reflect on professional situations both emotionally and rationally. Central to these discussions was the creation of a safe, trustworthy, and collegial environment, aligning with evidence in the literature [ 34 ]. Clinical supervision provided a platform to share resources (information, knowledge, and skills) and address issues while offering mutual support [ 35 ]. The emergence of COVID-19 has stressed the significance of peer group clinical supervision and support for the nursing/midwifery workforce [ 36 ], highlighting the need to help nurses/midwifes preserve their well-being and participate in collaborative problem-solving. COVID-19 impacted and disrupted clinical supervision frequency, duration and access [ 37 ]. What was evident during COVID-19 was the stress and need for support for staff and given the restorative or supportive functions of clinical supervision it is a mechanism of support. However, clinical supervisors need support themselves to be able to better meet the supervisee’s needs [ 38 ].

The value of peer group clinical supervision in nurturing a conducive working environment cannot be overstated, as it indorses the understanding and adherence to workplace policies by empowering supervisees to understand the importance and rationale behind these policies [ 39 ]. This becomes vital in a continuously changing healthcare landscape, where guidelines and policies may be subject to change, especially in response to situations such as COVID-19. In an era characterised by international workforce mobility and a shortage of healthcare professionals, a supportive and positive working environment through the provision of peer group clinical supervision can positively influence staff retention [ 40 ], enhance job satisfaction [ 41 ], and mitigate burnout [ 42 ]. A critical aspect of the peer group clinical supervision process concerns providing staff the opportunity to reflect, step back, problem-solve and generate solutions. This, in turn, ensures critical thinking and evaluation within clinical supervision, focusing on understanding the issues and context, and problem-solving to draw constructive lessons for the future [ 30 ]. Research has determined a link between clinical supervision and improvements in the quality and standards of care [ 31 ]. Therefore, peer group clinical supervision plays a critical role in enhancing patient safety by nurturing improved communication among staff, facilitating reflection, promoting greater self-awareness, promoting the exchange of ideas, problem-solving, and facilitating collective learning from shared experiences.

Starting a group arose as a foundational aspect emphasised in this study. The creation of the environment through establishing ground rules, building relationships, fostering trust, displaying respect, and upholding confidentiality was evident. Vital to this process is the recruitment of clinical supervisees and deciding the suitable group size, with a specific emphasis on addressing individuals’ inclination to engage, their knowledge and understanding of peer group clinical supervision, and dissipating any lack of awareness or misconceptions regarding peer group supervision. Furthermore, the educational training of peer group clinical supervisors and the support from external clinical supervisors played a vital role in the rollout and formation of peer group clinical supervision. The evidence stresses the significance of an open and safe environment, wherein supervisees feel secure and trust their supervisor. In such an environment, they can effectively reflect on practice and related issues [ 41 ]. This study emphasises that the effectiveness of peer group supervision is more influenced by the process than the content. Clinical supervisors utilised the process to structure their sessions, fostering energy and interest to support their peers and cultivate new insights. For peer group clinical supervision to be effective, regularity is essential. Meetings should be scheduled in advance, allocate protected time, and take place in a private space [ 35 ]. While it is widely acknowledged that clinical supervisors need to be experts in their professional field to be credible, this study highlights that the crucial aspects of supervision lie in the quality of the relationship with the supervisor. The clinical supervisor should be supportive, caring, open, collaborative, sensitive, flexible, helpful, non-judgmental, and focused on tacit knowledge, experiential learning, and providing real-time feedback.

Critical to the success of peer group clinical supervision is the endorsement and support from management, considering the organisational culture and attitudes towards the practice of clinical supervision as an essential factor [ 43 ]. This support and buy-in are necessary at both the management and individual levels [ 28 ]. The primary obstacles to effective supervision often revolve around a lack of time and heavy workloads [ 44 ]. Clinical supervisors frequently struggle to find time amidst busy environments, impacting the flexibility and quality of the sessions [ 45 ]. Time constraints also limit the opportunity for reflection within clinical supervision sessions, leaving supervisees feeling compelled to resolve issues on their own without adequate support [ 45 ]. Nevertheless, time-related challenges are not unexpected, prompting a crucial question about the value placed on clinical supervision and its integration into the culture and fabric of the organisation or profession to make it a customary practice. Learning from experiences like those during the COVID-19 pandemic has introduced alternative ways of working, and the use of technology (such as Zoom, Microsoft Teams, Skype) may serve as a means to address time, resource, and travel issues associated with clinical supervision.

Despite clinical supervision having a long international history, persistent misconceptions require attention. Some of these include not considering clinical supervision a priority [ 46 ], perceiving it as a luxury [ 41 ], deeming it self-indulgent [ 47 ], or viewing it as mere casual conversation during work hours [ 48 ]. A significant challenge lies in the lack of a shared understanding regarding the role and purpose of clinical supervision, with past perceptions associating it with surveillance and being monitored [ 48 ]. These negative connotations often result in a lack of engagement [ 41 ]. Without encouragement and recognition of the importance of clinical supervision from management or the organisation, it is unlikely to become embedded in the organisational culture, impeding its normalisation [ 39 ].

In this study, some peer group clinical supervisors expressed feelings of being impostors and believed they lacked the knowledge, skills, and training to effectively fulfil their roles. While a deficiency in skills and competence are possible obstacles to providing effective clinical supervision [ 49 ], the peer group clinical supervisors in this study did not report such issues. Instead, their concerns were more about questioning their ability to function in the role of a peer group clinical supervisor, especially after a brief training program. The literature acknowledges a lack of training where clinical supervisors may feel unprepared and ill-equipped for their role [ 41 ]. To address these challenges, clinical supervisors need to be well-versed in professional guidelines and ethical standards, have clear roles, and understand the scope of practice and responsibilities associated with being a clinical supervisor [ 41 ].

The support provided by external clinical supervisors and the peer group clinical supervision sessions played a pivotal role in helping peer group clinical supervisors ease into their roles, gain experiential learning, and enhance their facilitation skills within a supportive structure. Educating clinical supervisors is an investment, but it should not be a one-time occurrence. Ongoing external clinical supervision for clinical supervisors [ 50 ] and continuous professional development [ 51 ] are crucial, as they contribute to the likelihood of clinical supervisors remaining in their roles. However, it is important to interpret the results of this study with caution due to the small sample size in the survey. Generalising the study results should be approached with care, particularly as the study was limited to two regions in Ireland. However, the addition of qualitative data in this mixed-methods study may have helped offset this limitation.

This study highlights the numerous advantages of peer group clinical supervision at individual, service, organisational, and patient/client levels. Success hinges on addressing the initial lack of awareness and misconceptions about peer group clinical supervision by creating the right environment and establishing ground rules. To unlock the full potential of peer group clinical supervision, it is imperative to secure management and organisational support for staff release. More crucially, there is a need for valuing and integrating peer group clinical supervision into nursing and midwifery education and practice. Making peer group clinical supervision accessible to all grades of nurses and midwives across various healthcare services is essential, necessitating strategic planning to tackle capacity and sustainability challenges.

Data availability

Data are available from the corresponding author upon request owing to privacy or ethical restrictions.

Zelenikova R, Gurkova E, Friganovic A, Uchmanowicz I, Jarosova D, Ziakova K, Plevova I, Papastavrou E. Unfinished nursing care in four central European countries. J Nurs Manage. 2020;28(8):1888–900. https://doi.org/10.1111/jonm.12896 .

Article   Google Scholar  

Department of Health, Office of the Chief Nursing Officer. Position paper 1: values for nurses and midwives in Ireland. Dublin: The Stationery Office; 2016.

Google Scholar  

Cummings J, Bennett V. Developing the culture of compassionate care: creating a new vision for nurses, midwives and care-givers. London: Department of Health; 2012.

Both-Nwabuwe JM, Dijkstra MT, Klink A, Beersma B. Maldistribution or scarcity of nurses: the devil is in the detail. J Nurs Manage. 2018;26(2):86–93. https://doi.org/10.1111/jonm.12531 .

Squires A, Jylha V, Jun J, Ensio A, Kinnunen J. A scoping review of nursing workforce planning and forecasting research. J Nurs Manage. 2017;25:587–96. https://doi.org/10.1111/jonm.12510 .

Sasso L, Bagnasco A, Catania G, Zanini M, Aleo G, Watson R. Push and pull factors of nurses’ intention to leave. J Nurs Manage. 2019;27:946–54. https://doi.org/10.1111/jonm.12745 .

Gea-Caballero V, Castro-Sánchez E, Díaz‐Herrera MA, Sarabia‐Cobo C, Juárez‐Vela R, Zabaleta‐Del Olmo E. Motivations, beliefs, and expectations of Spanish nurses planning migration for economic reasons: a cross‐sectional, web‐based survey. J Nurs Scholarsh. 2019;51(2):178–86. https://doi.org/10.1111/jnu.12455 .

Article   PubMed   Google Scholar  

Cutcliffe J, Sloan G, Bashaw M. A systematic review of clinical supervision evaluation studies in nursing. Int J Ment Health Nurs. 2018;27:1344–63. https://doi.org/10.1111/inm.12443 .

Snowdon DA, Hau R, Leggat SG, Taylor NF. Does clinical supervision of health professionals improve patient safety? A systematic review and meta-analysis. Int J Qual Health C. 2016;28(4):447–55. https://doi.org/10.1093/intqhc/mzw059 .

Turner J, Hill A. Implementing clinical supervision (part 1): a review of the literature. Ment Health Nurs. 2011;31(3):8–12.

Dilworth S, Higgins I, Parker V, Kelly B, Turner J. Finding a way forward: a literature review on the current debates around clinical supervision. Contemp Nurse. 2013;45(1):22–32. https://doi.org/10.5172/conu.2013.45.1.22 .

Buss N, Gonge H. Empirical studies of clinical supervision in psychiatric nursing: a systematic literature review and methodological critique. Int J Ment Health Nurs. 2009;18(4):250–64. https://doi.org/10.1111/j.1447-0349.2009.00612.x .

Pollock A, Campbell P, Deery R, Fleming M, Rankin J, Sloan G, Cheyne H. A systematic review of evidence relating to clinical supervision for nurses, midwives and allied health professionals. J Adv Nurs. 2017;73(8):1825–37. https://doi.org/10.1111/jan.13253 .

Snowdon DA, Leggat SG, Taylor NF. Does clinical supervision of healthcare professionals improve effectiveness of care and patient experience: a systematic review. BMC Health Serv Res. 2017;17(1):1–11. https://doi.org/10.1186/s12913-017-2739-5 .

Kühne F, Maas J, Wiesenthal S, Weck F. Empirical research in clinical supervision: a systematic review and suggestions for future studies. BMC Psychol. 2019;7(1):1–11. https://doi.org/10.1186/s40359-019-0327-7 .

Snowdon DA, Sargent M, Williams CM, Maloney S, Caspers K, Taylor NF. Effective clinical supervision of allied health professionals: a mixed methods study. BMC Health Serv Res. 2020;20(1):1–11. https://doi.org/10.1186/s12913-019-4873-8 .

Borders LD. Dyadic, triadic, and group models of peer supervision/consultation: what are their components, and is there evidence of their effectiveness? Clin Psychol. 2012;16(2):59–71.

Health Service Executive. Guidance document on peer group clinical supervision. Mayo: Nursing and Midwifery Planning and Development Unit Health Service Executive West Mid West; 2023.

Sharma A, Minh Duc NT, Lam Thang L, Nam T, Ng NH, Abbas SJ, Huy KS, Marušić NT, Paul A, Kwok CL. Karamouzian, M. A consensus-based checklist for reporting of survey studies (CROSS). J Gen Intern Med. 2021;36(10):3179–87. https://doi.org/10.1007/s11606-021-06737-1 .

Article   PubMed   PubMed Central   Google Scholar  

O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;899:1245–51. https://doi.org/10.1097/ACM.0000000000000388 .

Winstanley J, White E. The MCSS-26©: revision of the Manchester Clinical Supervision Scale© using the Rasch Measurement Model. J Nurs Meas. 2011;193(2011):160–78. https://doi.org/10.1891/1061-3749.19.3.160 .

Colorafi KJ, Evans B. Qualitative descriptive methods in health science research. HERD-Health Env Res. 2016;9:16–25. https://doi.org/10.1177/1937586715614171 .

Agnew T, Vaught CC, Getz HG, Fortune J. Peer group clinical supervision program fosters confidence and professionalism. Prof Sch Couns. 2000;4(1):6–12.

Mc Carthy V, Goodwin J, Saab MM, Kilty C, Meehan E, Connaire S, O’Donovan A. Nurses and midwives’ experiences with peer-group clinical supervision intervention: a pilot study. J Nurs Manage. 2021;29:2523–33. https://doi.org/10.1111/jonm.13404 .

Rothwell C, Kehoe A, Farook SF, Illing J. Enablers and barriers to effective clinical supervision in the workplace: a rapid evidence review. BMJ Open. 2021;119:e052929. https://doi.org/10.1136/bmjopen-2021-052929 .

Francis A, Bulman C. In what ways might group clinical supervision affect the development of resilience in hospice nurses. Int J Palliat Nurs. 2019;25:387–96. https://doi.org/10.12968/ijpn.2019.25.8.387 .

Chircop Coleiro A, Creaner M, Timulak L. The good, the bad, and the less than ideal in clinical supervision: a qualitative meta-analysis of supervisee experiences. Couns Psychol Quart. 2023;36(2):189–210. https://doi.org/10.1080/09515070.2021.2023098 .

Stacey G, Cook G, Aubeeluck A, Stranks B, Long L, Krepa M, Lucre K. The implementation of resilience based clinical supervision to support transition to practice in newly qualified healthcare professionals. Nurs Educ Today. 2020;94:104564. https://doi.org/10.1016/j.nedt.2020.104564 .

Feerick A, Doyle L, Keogh B. Forensic mental health nurses’ perceptions of clinical supervision: a qualitative descriptive study. Issues Ment Health Nurs. 2021;42:682–9. https://doi.org/10.1080/01612840.2020.1843095 .

Corey G, Haynes RH, Moulton P, Muratori M. Clinical supervision in the helping professions: a practical guide. Alexandria, VA: American Counseling Association; 2021.

Sturman N, Parker M, Jorm C. Clinical supervision in general practice training: the interweaving of supervisor, trainee and patient entrustment with clinical oversight, patient safety and trainee learning. Adv Health Sci Educ. 2021;26:297–311. https://doi.org/10.1007/s10459-020-09986-7 .

Alfonsson S, Parling T, Spännargård Å, Andersson G, Lundgren T. The effects of clinical supervision on supervisees and patients in cognitive behavioral therapy: a systematic review. Cogn Behav Therapy. 2018;47(3):206–28. https://doi.org/10.1080/16506073.2017.1369559 .

Coelho M, Esteves I, Mota M, Pestana-Santos M, Santos MR, Pires R. Clinical supervision of the nurse in the community to promote quality of care provided by the caregiver: scoping review protocol. Millenium J Educ Technol Health. 2022;2:83–9. https://doi.org/10.29352/mill0218.26656 .

Toros K, Falch-Eriksen A. Structured peer group supervision: systematic case reflection for constructing new perspectives and solutions. Int Soc Work. 2022;65:1160–5. https://doi.org/10.1177/0020872820969774 .

Bifarin O, Stonehouse D. Clinical supervision: an important part of every nurse’s practice. Brit J Nurs. 2017;26(6):331–5. https://doi.org/10.12968/bjon.2017.26.6.331 .

Turner J, Simbani N, Doody O, Wagstaff C, McCarthy-Grunwald S. Clinical supervision in difficult times and at all times. Ment Health Nurs. 2022;42(1):10–3.

Martin P, Tian E, Kumar S, Lizarondo L. A rapid review of the impact of COVID-19 on clinical supervision practices of healthcare workers and students in healthcare settings. J Adv Nurs. 2022;78:3531–9. https://doi.org/10.1111/jan.15360 .

van Dam M, van Hamersvelt H, Schoonhoven L, Hoff RG, Cate OT, Marije P. Hennus. Clinical supervision under pressure: a qualitative study amongst health care professionals working on the ICU during COVID-19. Med Edu Online. 2023;28:1. https://doi.org/10.1080/10872981.2023.2231614 .

Martin P, Lizarondo L, Kumar S, Snowdon D. Impact of clinical supervision on healthcare organisational outcomes: a mixed methods systematic review. PLoS ONE. 2021;1611:e0260156. https://doi.org/10.1371/journal.pone.0260156 .

Article   CAS   Google Scholar  

Hussein R, Salamonson Y, Hu W, Everett B. Clinical supervision and ward orientation predict new graduate nurses’ intention to work in critical care: findings from a prospective observational study. Aust Crit Care. 2019;325:397–402. https://doi.org/10.1016/j.aucc.2018.09.003 .

Love B, Sidebotham M, Fenwick J, Harvey S, Fairbrother G. Unscrambling what’s in your head: a mixed method evaluation of clinical supervision for midwives. Women Birth. 2017;30:271–81. https://doi.org/10.1016/j.wombi.2016.11.002 .

Berry S, Robertson N. Burnout within forensic psychiatric nursing: its relationship with ward environment and effective clinical supervision? J Psychiatr Ment Health Nurs. 2019;26:7–8. https://doi.org/10.1111/jpm.12538 .

Markey K, Murphy L, O’Donnell C, Turner J, Doody O. Clinical supervision: a panacea for missed care. J Nurs Manage. 2020;28:2113–7. https://doi.org/10.1111/jonm.13001 .

Brody AA, Edelman L, Siegel EO, Foster V, Bailey DE Jr., Bryant AL, Bond SM. Evaluation of a peer mentoring program for early career gerontological nursing faculty and its potential for application to other fields in nursing and health sciences. Nurs Outlook. 2016;64(4):332–8. https://doi.org/10.1016/j.outlook.2016.03.004 .

Bulman C, Forde-Johnson C, Griffiths A, Hallworth S, Kerry A, Khan S, Mills K, Sharp P. The development of peer reflective supervision amongst nurse educator colleagues: an action research project. Nurs Educ Today. 2016;45:148–55. https://doi.org/10.1016/j.nedt.2016.07.010 .

Pack M. Unsticking the stuckness’: a qualitative study of the clinical supervisory needs of early-career health social workers. Brit J Soc Work. 2015;45:1821–36. https://doi.org/10.1093/bjsw/bcu069 .

Bayliss J. Clinical supervision for palliative care. London: Quay Books; 2006.

Kenny A, Allenby A. Implementing clinical supervision for Australian rural nurses. Nurs Educ Pract. 2013;13(3):165–9. https://doi.org/10.1016/j.nepr.2012.08.009 .

MacLaren J, Stenhouse R, Ritchie D. Mental health nurses’ experiences of managing work-related emotions through supervision. J Adv Nurs. 2016;72:2423–34. https://doi.org/10.1111/jan.12995 .

Wilson HM, Davies JS, Weatherhead S. Trainee therapists’ experiences of supervision during training: a meta-synthesis. Clinl Psychol Psychother. 2016;23:340–51. https://doi.org/10.1002/cpp.1957 .

Noelker LS, Ejaz FK, Menne HL, Bagaka’s JG. Factors affecting frontline workers’ satisfaction with supervision. J Aging Health. 2009;21(1):85–101. https://doi.org/10.1177/0898264308328641 .

Download references

Acknowledgements

The research team would like to thank all participants for their collaboration, the HSE steering group members and Carmel Hoey, NMPDU Director, HSE West Mid West, Dr Patrick Glackin, NMPD Area Director, HSE West, Annette Cuddy, Director, Centre of Nurse and Midwifery Education Mayo/Roscommon; Ms Ruth Hoban, Assistant Director of Nursing and Midwifery (Prescribing), HSE West; Ms Annette Connolly, NMPD Officer, NMPDU HSE West Mid West.

The authors declare that there are no sources of funding associated with this paper.

Author information

Authors and affiliations.

Department of Nursing and Midwifery, Health Research Institute, University of Limerick, Limerick, Ireland

Owen Doody, Kathleen Markey, Claire O. Donnell & Louise Murphy

Department of Nursing and Midwifery, Sheffield Hallam University, Sheffield, UK

James Turner

You can also search for this author in PubMed   Google Scholar

Contributions

OD: Conceptualization, Methodology, Formal analysis, Investigation, Writing - Original Draft, Writing - Review & Editing, Project administration, Funding acquisition. COD: Methodology, Formal analysis, Investigation, Writing - Original Draft, Writing - Review & Editing, Funding acquisition. KM: Methodology, Formal analysis, Investigation, Writing - Original Draft, Writing - Review & Editing, Funding acquisition. JT: Methodology, Formal analysis, Writing - Original Draft, Writing - Review & Editing. LM: Methodology, Formal analysis, Investigation, Writing - Original Draft, Writing - Review & Editing, Funding acquisition.

Corresponding author

Correspondence to Owen Doody .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by two health service institutional review boards University Hospital Limerick (Ref: 091/19) and Galway University Hospitals (Ref: C.A. 2199). The study was conducted in accordance with the principles of the Declaration of Helsinki and all study details were fully disclosed to participants, who were assured of the voluntary nature of participation and withdrawal. The study questionnaires were coded, and identities were not disclosed to guarantee participants’ anonymity and all participants provided written informed consent before interviews.

Consent for publication

No identifying images or other personal or clinical details of participants are presented in this paper that would compromise anonymity and all participants were aware and informed through the Participant Information Leaflet that the data collected may be reported through article and/or conference publications and this was reiterated during the qualitative data collection stage.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Doody, O., Markey, K., Turner, J. et al. Clinical supervisor’s experiences of peer group clinical supervision during COVID-19: a mixed methods study. BMC Nurs 23 , 612 (2024). https://doi.org/10.1186/s12912-024-02283-3

Download citation

Received : 04 January 2024

Accepted : 22 August 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s12912-024-02283-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical supervision

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 03 September 2024

Study of dog population dynamics and rabies awareness in Thailand using a school-based participatory research approach

• Weerakorn Thichumpa 1 ,
• Anuwat Wiratsudakul 2 ,
• Sarin Suwanpakdee 2 ,
• Chayanin Sararat 3 ,
• Charin Modchang 3 , 4 ,
• Setha Pan-ngum 5 ,
• Nakornthip Prompoon 5 ,
• Onpawee Sagarasaeranee 6 ,
• Sith Premashthira 6 ,
• Weerapong Thanapongtharm 6 ,
• Arun Chumkaeo 7 &
• Wirichada Pan-ngum 1 , 8  

Scientific Reports volume  14 , Article number:  20477 ( 2024 ) Cite this article

Metrics details

• Epidemiology
• Health care
• Risk factors

Rabies is a neglected disease primarily related to dog-mediated transmission to humans. Accurate dog demographic and dynamic data are essential for effectively planning and evaluating population management strategies when designing interventions to prevent rabies. However, in Thailand, longitudinal survey data regarding dog population size are scarce. A school-based participatory research (SBPR) approach was conducted to survey owned dogs for one year in four high-risk provinces (Chiang Rai, Surin, Chonburi, and Songkhla) of Thailand, aiming to understand dog population dynamics and raise awareness about rabies. ‘Pupify’ mobile application was developed to collect data on dog population and observe the long-term population dynamics in this study. At the end of the data collection period, telephone interviews were conducted to gain insight into contextual perceptions and awareness regarding both animal and human rabies, as well as the social responsibility of dog owners in disease prevention and control. Among 303 high school students who registered in our study, 218 students reported at least one update of their dog information throughout the one-year period. Of 322 owned dogs from our survey, the updates of dog status over one year showed approximately 7.5 newborns per 100-dog-year, while deaths and missing dogs were 6.2 and 2.7 per 100-dog-year, respectively. The male to female ratio was approximately 1.8:1. Twenty-three students (10%) voluntarily participated and were interviewed in the qualitative study. The levels of rabies awareness and precautions among high-school students were relatively low. The high dropout rate of the survey was due to discontinuity in communication between the researcher and the students over the year. In conclusion, this study focused on using the SBPR approach via mobile application to collect data informing dog population dynamics and raising awareness regarding rabies in Thailand Other engaging platforms (e.g. Facebook, Instagram, Twitter, and other popular applications) is necessary to enhance communication and engagement, thereby sustaining and maintaining data collection. Further health education on rabies vaccination and animal-care practices via social media platforms would be highly beneficial. For sustainable disease control, engaging communities to raise awareness of rabies and increase dog owners’ understanding of their responsibilities should be encouraged.

Introduction

Population demographics are important baseline data necessary for the study of infectious diseases. Human population data are available in most settings. For animal populations, however, demographic information is very limited in several countries and often only available for specific cohorts or studies. In Thailand, nationwide dog surveys are conducted by local government organizations once or twice a year and reported to a web-based reporting system, “ThaiRabies.net”, which has been updated to “Rabies One Data” since 2021 1 . These surveys require considerable human resources, while the quality of data can vary from province to province depending on the management and training of local staff teams to process and manage data 2 . Here, we proposed an innovative way to conduct dog surveys using a school-based participatory research (SBPR) as a part of community-based participatory research (CBPR), an approach to research that involves collective, reflective, and systematic inquiry in which researchers and community stakeholders engage as equal partners in all steps of the research process, with the goal of educating, improving practice, or bringing about social change 3 , 4 . We implemented the SBPR approach to perform a dog population survey among high school students in Thailand, using a mobile-phone application. This alternative approach relies on a self-reporting system for dog owners. This can be done through a mobile application developed for data collection. This approach was hoped to provide solution of a long-term data collection with lower cost to the government sectors, as well as promote community participations, raising awareness and responsibility among owners to register, monitor, and care for their dogs.

Dog ownership issues are critical for the design of rabies vaccination campaigns, especially in developing countries, including Thailand 5 . In many high-income settings, owners are responsible for properly confining their dogs and facilitating their vaccination against rabies. In Thailand, dog-keeping practices and duties of responsible ownership vary depending on the cultural setting 6 . There is an increasing evidence that most free-roaming dogs are owned and accessible for rabies prophylaxis 7 , 8 , 9 ; moreover, unvaccinated owned dogs have been affected by rabies 2 . Nevertheless, many owners cannot afford to pay for vaccination and other veterinary care for their own dogs 10 , 11 . Thus, many people rely on free, mass vaccination campaigns against rabies, provided by the government or non-governmental organizations (NGOs). In addition, limited access to dog vaccination can potentially reduce effective vaccination coverage, particularly if the proportion of unowned dogs is large. Dog movement patterns can also play a role in rabies epidemiology 12 . Dog confinement has been studied and implemented in some countries as a control measure for rabies 13 , 14 , 15 .

In Thailand, rabies is a notifiable condition, however it is not compulsory to report suspected rabies exposure in humans 16 . Both dog and human vaccination guidelines from the World Health Organization (WHO) and the World Organization for Animal Health (WOAH), recommend a comprehensive strategy to eradicate dog-mediated rabies 17 , 18 , 19 . The strategy highlights the importance of mass dog vaccination campaigns (aiming for at least 70% coverage) and the implementation of effective dog population control measures (e.g. sterilization), which have been optimized for rabies prevention and control 16 , 17 , 18 , 20 . Human rabies in Thailand has been prevented and controlled by policy promulgated since 1992. Rabies cases have decreased because of schemes including mass dog vaccination and sterilization. Although human rabies in Thailand has gradually declined, animal rabies has been generally increasing over the past ten years 2 . In 2020, there were 209 cases of rabid dogs reported in Thailand and three human deaths due to rabies. Rabies is most prevalent in the provinces of Chonburi, Songkhla, and Surin, while Chiang Rai has found high positive detection of rabid animal cases in 2018 21 , 22 . The control of rabies in animals is challenging, as the disease can be transmitted throughout the year and therefore surveillance and control of animal carriers are urgently required 20 . As for the Thai government’s policy and guideline (based on WHO & WOAH) for high-risk areas, ring vaccination is currently implemented for controlling and preventing rabies outbreaks, while sterilization is a long-term solution to control number of dog population, reducing contacts among dogs and between human and dogs. Both vaccination and sterilization are hopeful for improve management of dog bites 22 .

Although the database of dogs has been significantly improved following the introduction of dog survey reporting to ThaiRabies.net by local government organizations, the system still relies solely on the public health sectors. Moreover, data consistency remains an issue due to technical problems within the system and incomplete data entry. Here, we introduced a novel method for owned-dog data collection, using the SBPR approach. Information about dog population dynamics is essential for analyzing population and disease prediction and can act as baseline data for dog population management plans. The exploration and identification of dog population ecosystems and dynamics are required as a framework to effectively plan and evaluate population management and interventions to prevent rabies 8 . In addition, the introduction of an approach to our dog survey among school-age children could be beneficial in terms of generating awareness of animal-care practices, disease, and the development of a research mindset.

Countries in Southeast Asia are among the top users of mobile phones globally. In 2020, total population of Thailand were approximately 65.42 million 23 . The number of smartphone users in Thailand reached 53.57 million, with around 60 million predicted by 2026, due both to increases in the Thai population and internet penetration 24 . Self-reported data collection via mobile phones can be of use when conducting large-scale surveys, with the affordability and availability of mobile phones and wireless networks making them a viable alternative to traditional methods 25 . However, it is important to consider various aspects involved in the development and implementation of mobile phone data collection. For example, ensuring usability and user acceptance of the data collection system will help motivate survey participants to stay with the project and continue to provide high-quality data. Server authentication through the use of properly configured certificates will help deal with threats of data submission to a malicious server, which can increase users’ confidence in data security 26 .

Our study proposed an initial effort to conduct a long-term survey based on dog owners’ awareness and participation. The dog population dynamics data were analyzed and visualized. In addition, the qualitative study was performed on 10% of the survey participants who volunteered to do the interview on knowledge of rabies, social responsibility, community engagement and research orientation. The data collection tools and methods were assessed and further improvements when using this approach were proposed.

Dog population survey

School and participant demographics.

In the survey via ‘Pupify’ mobile application, 303 high-school students registered through the mobile application for our study. There were 29.8%, 28.9%, 27.1% and 14.2% from a school in Chonburi, Surin, Chiang Rai, and Songkhla provinces, respectively; most were female participants (72.9%) (Table 1 ). Of 303 registrations, 218 participants actually submitted at least one update of their dogs into the system over the one-year study. However, the number of participants continued submitting the monthly dog updates dropped to 46, 63 and 43 after 6 months, 9 months and by the end of one year, respectively. The number of students giving the completely one-year updates was 43 or 20% of total participants from the start (Fig.  1 ).

A number of participants’ responses in a 3-month period during the study year.

Dog demographics and dog population dynamics

Overall, 322 owned dogs were reported during the study period. More than half were male dogs (65.0%). Owned dogs were divided into three age groups based on owners′ identification: birth to 1 year (28.3%), aged between > 1 and 8 years (57.1%), and aged > 8 years (14.6%). These age classes were used to represent three groups of dogs, puppy, adult, and elderly. Most owned dogs were reported in Surin province (35.4%), followed by Chiang Rai (28.5%), Chonburi (22.7%), Songkhla (12.1%), and others where owners dwelling in adjacent areas (1.2%). In addition, 24 new-born puppies were reported, while there were 20 deaths (e.g. caused by dog illness, bite, fight, accident, and culling) and 9 missing dogs reported. These numbers correspond to the estimated birth, death, and missing rates of 7.5, 6.2, and 2.7 per 100 dog-years, respectively. Based on the self-reporting system, 40.1% of the dogs had been vaccinated against rabies and 12.4% had been sterilized (Table 2 ).

Qualitative study

Dog owner characteristics.

A total of 23 high-school students, all aged 17 years, voluntarily participated in our interview (see Supplementary Table 3 ). There were students from all three levels of participation, including registration only (17.4%, n = 4), partially updated data (39.1%, n = 9), and fully updated data (43.5%, n = 10). Although all schools from four provinces were represented, more than half of the participants were from Chonburi province (52.2%).

Extensive knowledge and dog rabies awareness

Most participants (91.3%, n = 21) strongly agreed that rabies was fatal, resulting in death in both humans and dogs. One participant noted, “I learned from the news on TV that human infections result in a hundred percent mortality” . However, 52.1% of the participants (n = 12) reported that they were either unaware of or did not follow rabies situations locally. This indicated that while most participants are aware of rabies, they do not necessarily stay informed about local rabies situations. One participant said, “ I have very little experience of rabies disease. I have not seen the real case before and have not followed the disease situation. At school, there is minimal information for us to research more about rabies. Sometimes, external health staff came to educate us about health at school but didn’t focus on rabies” . While a majority (65.2%, n = 15) of participants considered that only cats and dogs were reservoirs for rabies, a larger proportion (78.2%, n = 18) were unsure whether there were other animal reservoirs. This result indicated that most participants were unaware that other mammals can also get infected with rabies. From the interviews, some participants made statements such as “I think it mainly comes from dogs and cats, unlikely to be other species” and “Most cases are infected from stray dogs, perhaps also from rabbits and monkeys” . In addition, 65.0% (n = 15) of participants mainly received information about rabies from social media and other online sources, while the remaining participants obtained information from other sources, including schools (such as our project visit), television and news, community announcements, medical providers, parents, and relatives.

Rabies precautions and caring for owned dogs

Most participants (87.0%, n = 20) stated that avoiding contact with stray dogs can help to prevent rabies infection. Also, 52.1% (n = 12) suggested that owned dogs should be vaccinated annually against rabies. Dog confinement was reported by most owners (87.0%, n = 20) as a way to control and limit their dogs’ contact with humans or animals. One participant said, “I keep my dog only in my house to avoid contacting with people and other dogs” and “My dog is always leashed all the time and I don’t allow other dogs nearby my dog when it is outside” . According to this, half of them (52.1%, n = 12) trusted their dogs, with 80–100% confidence due to annually vaccination and not allowing dogs outside. One participant said, “Some of my dogs are not yet vaccinated, we put the dogs to guard our properties in the factory area and sometimes outside dogs do come to visit” .

In terms of caring for owned dogs, participants reported how they managed their dog’s health (including regular health check-ups and visits to veterinarians when health issues were identified). The majority used the services of animal clinics (87.0%, n = 20), followed by animal hospitals (21.7%, n = 5), treatment by owners (21.7%, n = 5), and government veterinary services (13.0%, n = 3). However, one said, “I saw my aunt giving paracetamol to the dog when it was sick. I didn't agree with that and would have looked for more information or taken the dog to the vet instead” . This indicated that animal health education on the care of owned dogs should be enhanced, with information provided by specialists at animal service stations.

In the case of what happens to newborn puppies, participants identified two common situations: giving them away to others (65.2%, n = 15) and keeping the puppies themselves (39.1%, n = 9). In the mating season, most participants said they confined their dogs and did not allow them to breed with other dogs. One participant said, “I usually keep the dog in the house and sometimes use a lease to prevent dogs fighting”. Conversely, in the case of both neutered and non-neutered dogs, some participants still allowed their dogs to breed. Finally, the owners said they commonly observed their dog’s health status at feeding time (47.8%, n = 11); when they were sleeping (30.4%, n = 7) or playing (17.4%, n = 4); or when they observed any abnormality (17.4%, n = 4).

Obstacles, limitations, and motivations for joining in with school-based participatory research

Obstacles and limitations relating to the SBPR study mentioned by participants included forgetting to update their dog’s data (65.2%, n = 15), having school assignments and portfolios (30.4%, n = 7), having a part-time job (17.4%, n = 4), having personal works (17.4%, n = 4), having a poor internet connection (13.0%, n = 3), changing their smartphone (8.7%, n = 2), being unable to install the mobile application (4.3%, n = 1), and not interested in participating (4.3%, n = 1).

Conversely, participants reported some interesting advantages and motivations for why they participated in this study. Motivations included in the attainment of project certificates (60.9%, n = 14), followed by project rewards/gifts (34.8%, n = 8), research experience (13.0%, n = 3), dog care and follow-up (13.0%, n = 3), and rabies information (4.3%, n = 1). Other influences for joining the project mentioned included own self (65.2%, n = 15), project notification (13.0%, n = 3), project rewards (8.7%, n = 2), and support for school activities (4.3%, n = 1). After participated in this study, the main advantages given were mostly focused on caring for owned dogs, with regard to dog attention and care (69.6%, n = 16), observation of dog behavior (34.8%, n = 8), dog vaccine notification (17.4%, n = 4), and education (17.4%, n = 4). One mentioned that “In my opinion, the best thing I learned is to pay more attention to my dog. I observe my dogs more regularly and take care of them much better than earlier” .

Other suggestions from participants

Some participants suggested that they needed more information about rabies disease, its prevention and control, dog management, and dog vaccination. This could be added to the Pupify application, which was easily accessible for necessary information. Also, alternative sources of information should be considered, e.g., infographics and dog fan-pages on Facebook, Instagram, Twitter, or other popular social media platforms. One participant suggested, “I think having different channels for communication would help stimulate more interest in the work, for example, forming a ‘dog lovers’ group on social media” .

Here, we explored a new method to collect dog data via mobile application, a self-reporting system for dog owners, by focus initially on high school students who owned smartphones, which is in contrast with the conventional dog population census that is performed once or twice per year in Thailand by the government departments responsible for animal health. The key challenge to our design was the number of losses to follow-up. Our qualitative study revealed the main barriers to update dog dynamics data were due to some personal issues and technical reasons. A participant from the partial update group noted, “I gave regular updates until I changed my smart phone, I stopped updating the information completely” . One from the no-update group said, “I had difficulties installing the app and I think I am not disciplined enough to join this project anyway”. In addition, there was some feedback on the suitability of a mobile instant messaging app for data tracing. One participant suggested, “I prefer other channels of communication such as Instagram and Facebook because they are more convenient to me” .

Nevertheless, we estimated birth, death, and missing rates of 7.5, 6.2, and 2.7 per 100 dog-years, respectively. The male to female ratio was approximately 1.8:1. The variations in these rates and ratios among the studied provinces are noticeable (see Supplementary Tables 1 and 2 ). This could be due to different nature of owned dogs in different parts of Thailand. However, due to the relatively small sample size in our study, it would not be appropriate to perform any sub-analysis from this data. It is important to note that the majority of the data provided pertained to confined dogs (70.2%), which may not accurately reflect the uncertainty conditions of free-roaming dogs. Future dog censuses should include a focus on confined, free-roaming, and stray dogs to provide a more comprehensive representation of the overall dog population size. Observations in South Africa revealed that birth and death rates were 31.3–45.1 and 40.6–56.8 per 100 dog-years, respectively, while the male to female ratio was approximately 1.4–1.7:1 27 . A study in India estimated an annual per capita birth and death rate of 1.0 and 0.7, respectively, while the male to female ratio was approximately 1.4:1 28 . A sight–resight survey in Australia reported birth and death rates were approximately 2.4 and 1.7 dogs/dog-owning house/year, respectively, while the male to female ratio was approximately 1:1 7 . Compared with other studies (using different approaches to collect the data; including observational, sight-resight, and/or mark-recapture survey), births and deaths in our study were relatively low. However, the male to female ratio was in line with previous studies. Similarly to a previous study 6 , we found the proportions of dog-keeping approaches (i.e. confined or free-roaming) varied among the sites, with dogs usually confined in well-developed areas whereas free-roaming dogs were reported more frequently in remote areas.

Our study had some limitations. First, the survey was restricted to owned dogs. It would be helpful to collect similar data for stray dogs; however, to conduct a similar study of stray dogs in the Thai setting, individuals who take care of stray dogs, so called “local feeders”, must be identified 29 . Second, the participants only comprised high-school children of a specific age group, perhaps a broader target public population should be considered for future surveys. Furthermore, we simply used three reproductive age classes to represent puppy, adult, and elderly i.e. the exact dog ages as detailed classifications, i.e. puppy, juvenile, young adult, mature adult, senior, and geriatric, are not available in this study. Third, the 'Pupify' application was developed for Android phones only and required updates to remain compatible with the latest operating system versions. Fourth, there was a low number of one-year data completion among the participated students who owned a mobile phone. Because the participation was voluntary, unrelated to school nor teacher’s request. The study sites were distant from the central project location, notifications and encouragement communications were conducted solely via Line messaging application and telephone calls. This led to discontinuities in communication between the researchers and the students throughout the year. The barriers in our SBPR engagement were limitations of the mobile application platform, technical issues, personal reasons, and the lack of engagement of project through the teachers and/or schools. Further studies should consider site visits to enhance communication, encourage participation, and investigate any arising issues.

In accordance with “One Health” concepts, human health is closely connected to the health of animals and our shared environment, and research in this area should be a collaborative, multisectoral, and trans-disciplinary approach to achieve optimal health outcomes. We made considerable effort to use the SBPR approach in conducting this study. In addition, the initial motivation for study participation was primarily driven by the desire to achieve long-term goals and enhance their profiles for university enrollment. After participating, they also recognized considerable benefits in caring for their dogs and demonstrated a commitment to sustainable effort for better dog care. Although there was a low response rate among participants, we could remark that the main advantage concerning caring for owned dogs was initially successful based on participants’ perception. Most interviewees agreed that this study would encourage them to pay more regular attention to their dogs regarding their health, vaccinations, and rabies prevention. Our study demonstrated the importance of encouraging, among school-age children, early learning about the importance of disease prevention and awareness, together with community engagement and social responsibility for their future. Finally, it is important to note that the success of several research depends on effective data collections. However, this study has provided valuable lessons, demonstrating that engaging the general public, beyond researchers and experts, presents considerable challenges. Practical issues such as invitations, communications, cooperations, maintaining engagements, and overall participations should be carefully considered. We hope that the insights gained from our study with SBPR may be beneficial for further studies and similar contexts.

Conclusions

Using the SBPR approach for collecting dog population dynamics data among the high school students can be challenging. Additionally, this study was conducted with an initial effort to explore the potential of using SBPR for data collection. The primary objective aimed to propose extending the approach beyond student awareness to include general dog owners in further research. Implementing a suitable SBPR approach involves designing educational activities, training participants, conducting surveys, and engaging the community. This could lead to effective and sustained data collection while fostering community involvement and awareness in the future. Perception on the usefulness of the application and different social-media channels for communication should be considered for future development of data collecting tools and mobile application in order to provide higher incentive to participate and update dog information in a long-term. A low level of disease awareness among high school students was identified in the interviews, possibly due to insufficient information, both at school and in the media. It is critical to promote disease awareness through health education. Further studies using in-depth interviews should focus on enhancing rabies awareness, increasing owner responsibility, and supporting rabies prevention projects, as these factors are crucial for policymaking and effective public participation. Nevertheless, by conducting data collection using a new alternative approach among the students, it has clearly increased some awareness on the importance of animal welfare and provided some new experience of being part of a research for some students to reduce rabies among humans and animals.

Study sites and participants

This study was conducted between June 2018 and October 2019, in areas where rabies is endemic and where there is a high incidence of animal and human cases 30 . It formed part of a larger study conducted in Thailand between 2015 and 2018, which aimed to investigate the cultural and socioeconomic factors that contribute to rabies outbreaks in Thailand 31 . Four provinces were included: Chiang Rai province in the north, Surin province in the northeast, Chonburi province in the east, and Songkhla province in the south (Fig.  2 ). Based on the past five year report of rabies in Thailand 22 , 30 , we purposively surveyed high school students dwelling in high endemic areas among the four provinces. Inclusion criteria were: (1) students aged between 16 and 17 years who owned at least one dog and possessed a smartphone that used the Android operating system, and (2) volunteer students whose parents consented to their participation in the study. In this study, dog ownership was defined as those who owned or cared for at least one dog at the residence only. Students were eligible to voluntarily participate by registering dog data on the ‘Pupify’ application.

Maps showing; laboratory positive detection of rabies cases in animals in 2018 (Source: Thairabies.net: http://www.thairabies.net 1 ; and The four provinces included in the study: Chiang Rai, Surin, Chonburi, and Songkhla.

Data collection using the “Pupify”

‘Pupify’ mobile application was developed to collect long-term data on dog population numbers and dynamics from dog owners, feeders, and the general public. The ‘Pupify’ was developed by a group of university students from the Department of Computer Engineering, Chulalongkorn University 32 . The software architecture was three-tiered i.e. client, application server, and database server. The client section was initially constructed for Android OS using Java language. The application server was developed by using JavaScript which responded to user requests and monitored the types of data that should be recorded in the database server. All processes were tested accurately in both software testing and acceptance testing by developers and research team to ensure that the application can function in real settings.

In this study, the application was initially designed to target high-school students who have a smartphone and presumably have good knowledge of rabies. The application was developed in collaboration with the Department of Livestock Development (DLD), Ministry of Agriculture and Cooperatives Thailand, who are responsible for rabies control in Thailand. The application comprised three main sections: (i) demographic information about a dog’s owner, (ii) demographic information about dogs, and (iii) routine information updates and report management. The first and second sections were recorded in literal format once for each dog and owner upon registration. Monthly updates were required to follow-up on status of registered dogs, e.g. still alive, moved out, dead, vaccination status, and sterilization status. The participants were reminded to provide at least the monthly updates through the application and other channels of communication including Line messaging application and telephone calls with the researchers.

Qualitative study for the evaluation of participatory research

The second part of the study was conducted once the dog survey had been completed. This qualitative study aimed to explore in detail the knowledge, perceptions, and awareness of dog owners with regard to rabies in dogs and humans. Semi-structured interviews were used to collect the information. First, the participants from the survey were asked to voluntarily participate in the qualitative study by registering online to express their interest. To ensure a diversity of data, the research team purposively selected participants to include students whose duration of participation in the dog survey varied and those who attended different schools. Second, they were invited to participate in a one-to-one online interview with Thichumpa W. Each interview lasted for 15–30 min and was recorded. Informed consent was obtained from all participants’ parents. The interviews were conducted between March to May 2021.

The study protocol was approved by the ethical committees of Mahidol University Central Institutional Review Board (MU-CIRB 2019/157.0606; August 2019). Written informed consent was obtained from all high school students who participated in the research. All the methods were performed in accordance with relevant guidelines and regulations.

Data analyzes

Descriptive statistics were generated using SPSS version 23.0 33 . For the qualitative study, transcript data were evaluated by determining the frequency of answers given by interviewees and then coding keywords into pre-set themes 34 , including the theme of rabies knowledge, rabies awareness, caring for owned dog, perception about project, and other suggestions. The content analysis and thematic narrative approach were performed using QDA Miner Lite 35 .

Data availability

The data that support the findings of this study are available from the corresponding author, (WP), upon reasonable request.

Thai Rabies Net. Thai rabies report . http://www.thairabies.net/trn/ (2012).

Thanapongtharm, W. et al. Spatial distribution and population estimation of dogs in Thailand: Implications for rabies prevention and control. Front. Vet. Sci. 8 , 790701. https://doi.org/10.3389/fvets.2021.790701 (2021).

Article   PubMed   PubMed Central   Google Scholar  

Baum, F., MacDougall, C. & Smith, D. Participatory action research. J. Epidemiol. Community Health 60 , 854–857. https://doi.org/10.1136/jech.2004.028662 (2006).

Israel, B. A., Schulz, A. J., Parker, E. A. & Becker, A. B. Review of community-based research: Assessing partnership approaches to improve public health. Annu. Rev. Public Health 19 , 173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173 (1998).

Article   CAS   PubMed   Google Scholar  

Morters, M. K. et al. Participatory methods for the assessment of the ownership status of free-roaming dogs in Bali, Indonesia, for disease control and animal welfare. Prev. Vet. Med. 116 , 203–208. https://doi.org/10.1016/j.prevetmed.2014.04.012 (2014).

Kasempimolporn, S., Sichanasai, B., Saengseesom, W., Puempumpanich, S. & Sitprija, V. Stray dogs in Bangkok, Thailand: Rabies virus infection and rabies antibody prevalence. Dev. Biol. (Basel) 131 , 137–143 (2008).

CAS   PubMed   Google Scholar  

Hudson, E. G., Brookes, V. J. & Ward, M. P. Demographic studies of owned dogs in the Northern Peninsula Area, Australia, to inform population and disease management strategies. Aust. Vet. J. 96 , 487–494. https://doi.org/10.1111/avj.12766 (2018).

Tiwari, H. K., Robertson, I. D., O’Dea, M. & Vanak, A. T. Author correction: Demographic characteristics of free-roaming dogs (FRD) in rural and urban India following a photographic sight-resight survey. Sci. Rep. 10 , 3757. https://doi.org/10.1038/s41598-020-58147-8 (2020).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Wilson, P. J., Oertli, E. H., Hunt, P. R. & Sidwa, T. J. Evaluation of a postexposure rabies prophylaxis protocol for domestic animals in Texas: 2000–2009. J. Am. Vet. Med. Assoc. 237 , 1395–1401. https://doi.org/10.2460/javma.237.12.1395 (2010).

Article   PubMed   Google Scholar  

Knobel, D. L. et al. Rabies Scientific Basis of the Disease and Its Management Vol. 17, 591–615 (Elsevier Inc, 2013).

Google Scholar  

Arechiga Ceballos, N., Karunaratna, D. & Aguilar Setien, A. Control of canine rabies in developing countries: Key features and animal welfare implications. Rev. Sci. Tech. 33 , 311–321. https://doi.org/10.20506/rst.33.1.2278 (2014).

Raynor, B. et al. Movement patterns of free-roaming dogs on heterogeneous urban landscapes: Implications for rabies control. Prev. Vet. Med. 178 , 104978. https://doi.org/10.1016/j.prevetmed.2020.104978 (2020).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Smith, L. M. et al. The effectiveness of dog population management: A systematic review. Animals (Basel) https://doi.org/10.3390/ani9121020 (2019).

Ballantyne, K. C. Separation, confinement, or noises: What is scaring that dog?. Vet. Clin. N. Am. Small Anim. Pract. 48 , 367–386. https://doi.org/10.1016/j.cvsm.2017.12.005 (2018).

Article   Google Scholar  

Astorga, F., Poo-Munoz, D. A., Organ, J. & Medina-Vogel, G. Why let the dogs out? Exploring variables associated with dog confinement and general characteristics of the free-ranging owned-dog population in a peri-urban area. J. Appl. Anim. Welf. Sci. 25 , 311–325. https://doi.org/10.1080/10888705.2020.1820334 (2022).

Yurachai, O., Hinjoy, S. & Wallace, R. M. An epidemiological study of suspected rabies exposures and adherence to rabies post-exposure prophylaxis in Eastern Thailand, 2015. PLoS Negl. Trop. Dis. 14 , e0007248. https://doi.org/10.1371/journal.pntd.0007248 (2020).

The World Organization for Animal Health (WOAH). Rabies . https://www.woah.org/en/disease/rabies/#ui-id-2 (2024).

World Health Organization (WHO). Rabies . https://www.who.int/news-room/fact-sheets/detail/rabies#:~:text=Rabies%20infects%20mammals%2C%20including%20dogs,rabies%20is%20virtually%20100%25%20fatal (2022).

World Health Organization (WHO). One Health . https://www.who.int/news-room/fact-sheets/detail/one-health (2024).

Department of Disease Control-Ministry of Public Health. Rabies . https://ddc.moph.go.th/disease_detail.php?d=25 (2022).

Department of Disease Control & Ministry of Public Health. Rabies exposure report system (R36 database in Thai language) . http://r36.ddc.moph.go.th/r36/home or http://odpc9.ddc.moph.go.th/EOC/eoc.html (2022).

Department of Livestock Development-Ministry of Agriculture and Cooperatives. Rabies situation report in animals . https://dld.go.th/th/index.php/th/service-people/infographic-menu/64-hot-issue/rabies (2021).

National Statistical Office Thailand. Size and structure of the population report . https://www.nso.go.th/nsoweb/nso/statistics_and_indicators?order=&search=&impt_side=&impt_branch=300&impt_group=0&impt_subgroup=&year=2563&announcement_date= (2020).

Statista Research Department. Number of smartphone users in Thailand from 2017 to 2020 with a forecast through 2026 . https://www.statista.com/statistics/467191/forecast-of-smartphone-users-in-thailand/ (2021).

Tomlinson, M. et al. The use of mobile phones as a data collection tool: A report from a household survey in South Africa. BMC Med. Inform. Decis. Mak. 9 , 51. https://doi.org/10.1186/1472-6947-9-51 (2009).

Samaila, M. G., Neto, M., Fernandes, D. A. B., Freire, M. M. & Inácio, P. R. M. Challenges of securing Internet of Things devices: A survey. Secur. Privacy https://doi.org/10.1002/spy2.20 (2018).

Conan, A. et al. Population dynamics of owned, free-roaming dogs: Implications for rabies control. PLoS Negl. Trop. Dis. 9 , e0004177. https://doi.org/10.1371/journal.pntd.0004177 (2015).

Totton, S. C. et al. Stray dog population demographics in Jodhpur, India following a population control/rabies vaccination program. Prev. Vet. Med. 97 , 51–57. https://doi.org/10.1016/j.prevetmed.2010.07.009 (2010).

Komol, P., Sommanosak, S., Jaroensrisuwat, P., Wiratsudakul, A. & Leelahapongsathon, K. The spread of rabies among dogs in Pranburi District, Thailand: A metapopulation modeling approach. Front. Vet. Sci. 7 , 570504. https://doi.org/10.3389/fvets.2020.570504 (2020).

Bureau of Epidemiology, Department of Disease Control & Ministry of Public Health. Rabies annual reports (in Thai language) . https://ddc.moph.go.th/disease_detail.php?d=25 (2022).

Premashthira, S. et al. The impact of socioeconomic factors on knowledge, attitudes, and practices of dog owners on dog rabies control in Thailand. Front. Vet. Sci. 8 , 699352. https://doi.org/10.3389/fvets.2021.699352 (2021).

Luangcharoenpong, S. Application for Dog Census , Chulalongkorn University (2018).

IBM Corp. IBM SPSS Statistics for Windows [Computer software] . https://www.ibm.com/spss (2022).

Hsieh, H. F. & Shannon, S. E. Three approaches to qualitative content analysis. Qual. Health Res. 15 , 1277–1288. https://doi.org/10.1177/1049732305276687 (2005).

Provalis Research. QDA Miner Lite . https://provalisresearch.com/products/qualitative-data-analysis-software/freeware/ (2020).

Download references

Acknowledgements

We cordially thank all the high school students who participated in our surveys. We also thank Siwakorn Luengcharoenpong and the teams from the Department of Computer Engineering, Chulalongkorn University, Bangkok, Thailand for software development and consultation.

This study was funded by the National Science and Technology Development Agency (NSTDA), Thailand (Grant ID. P-18-51758) and the Disease Control Department, Ministry of Public Health, Thailand. In addition, this research was funded in part by the Wellcome Trust [220211]. For the purpose of Open Access, the authors have applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.

Author information

Authors and affiliations.

Department of Tropical Hygiene, Faculty of Tropical Medicine, Mahidol University, Bangkok, Thailand

Weerakorn Thichumpa & Wirichada Pan-ngum

Department of Clinical Sciences and Public Health, and the Monitoring and Surveillance, Center for Zoonotic Diseases in Wildlife and Exotic Animals, Faculty of Veterinary Science, Mahidol University, Nakhon Pathom, Thailand

Anuwat Wiratsudakul & Sarin Suwanpakdee

Biophysics Group, Department of Physics, Faculty of Science, Mahidol University, Bangkok, Thailand

Chayanin Sararat & Charin Modchang

Centre of Excellence in Mathematics, MHESI, Bangkok, Thailand

Charin Modchang

Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand

Setha Pan-ngum & Nakornthip Prompoon

Department of Livestock Development, Ministry of Agriculture and Cooperatives, Bangkok, Thailand

Onpawee Sagarasaeranee, Sith Premashthira & Weerapong Thanapongtharm

Songkhla Provincial Livestock Office, Muang, Songkhla, Thailand

Arun Chumkaeo

Mahidol Oxford Tropical Medicine Research Unit (MORU), Faculty of Tropical Medicine, Mahidol University, Bangkok, Thailand

Wirichada Pan-ngum

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization and Methodology: WK.T., W.P., C.M., A.W. and WP.T. Mobile application: S.P. and N.P. Survey and data collection: WK.T., S.S., C.S., O.S., S.PR., WP.T. and A.C. Formal analysis: WK.T. and W.P. Project administration and data management: WK.T. Writing–original draft: WK.T. and W.P. Writing–review & editing: All authors. The authors declare consent for publication.

Corresponding author

Correspondence to Wirichada Pan-ngum .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary tables., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Thichumpa, W., Wiratsudakul, A., Suwanpakdee, S. et al. Study of dog population dynamics and rabies awareness in Thailand using a school-based participatory research approach. Sci Rep 14 , 20477 (2024). https://doi.org/10.1038/s41598-024-71207-7

Download citation

Received : 10 May 2024

Accepted : 26 August 2024

Published : 03 September 2024

DOI : https://doi.org/10.1038/s41598-024-71207-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Dog population
• Rabies awareness
• School-based participatory research (SBPR)
• Public health

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Systematic Review
• Open access
• Published: 02 September 2024

Common data quality elements for health information systems: a systematic review

• Hossein Ghalavand 1 ,
• Saied Shirshahi 2 ,
• Alireza Rahimi 2 ,
• Zarrin Zarrinabadi 1 &
• Fatemeh Amani 3  

BMC Medical Informatics and Decision Making volume  24 , Article number:  243 ( 2024 ) Cite this article

Metrics details

Data quality in health information systems has a complex structure and consists of several dimensions. This research conducted for identify Common data quality elements for health information systems.

A literature review was conducted and search strategies run in Web of Knowledge, Science Direct, Emerald, PubMed, Scopus and Google Scholar search engine as an additional source for tracing references. We found 760 papers, excluded 314 duplicates, 339 on abstract review and 167 on full-text review; leaving 58 papers for critical appraisal.

Current review shown that 14 criteria are categorized as the main dimensions for data quality for health information system include: Accuracy, Consistency, Security, Timeliness, Completeness, Reliability, Accessibility, Objectivity, Relevancy, Understandability, Navigation, Reputation, Efficiency and Value- added. Accuracy, Completeness, and Timeliness, were the three most-used dimensions in literature.

Conclusions

At present, there is a lack of uniformity and potential applicability in the dimensions employed to evaluate the data quality of health information system. Typically, different approaches (qualitative, quantitative and mixed methods) were utilized to evaluate data quality for health information system in the publications that were reviewed. Consequently, due to the inconsistency in defining dimensions and assessing methods, it became imperative to categorize the dimensions of data quality into a limited set of primary dimensions.

Peer Review reports

Appropriate planning in the health sector relies on the existence of accurate data and the quality of the data must be continuously controlled. The World Health Organization has tried to ensure the quality of health data by providing a toolkit. This toolkit supports countries to assess and improve the quality of health data [ 1 , 2 ].

The existence of accurate, complete, and timely data plays an important role in health care management [ 3 , 4 , 5 ]. Data quality is often only considered a component of the effectiveness of health information systems, and hiding the value of data quality in other parts of the health field can lead to incorrect decision-making [ 6 , 7 , 8 , 9 ]. Previous studies have confirmed that data quality is a multidimensional concept. Data quality assessment requires familiarity with different subjective and objective criteria and both subjective perceptions of people and objective measurements of information must be addressed [ 10 , 11 ]. Qualitative evaluations of subjective data reflect the needs and experiences of stakeholders, and objective evaluations reflect the needs of managers and stakeholders [ 12 ].

Adverse effects on the quality of care, increasing costs, creating liability risks, and reducing the benefits of investing in health information systems can be identified as the negative effects of poor-quality data [ 13 , 14 , 15 , 16 ]. Defects in data quality can lead to incorrect diagnosis and intervention in health care [ 4 , 13 , 17 , 18 ]. The quality of healthcare depends on the existence of quality data, which ultimately leads to a significant impact on customer satisfaction [ 13 , 19 ].

Data quality in health information systems has a complex structure and consists of several dimensions and some critical factors performance such as environmental and organizational, technical and behavioral affected on data quality in health information system [ 20 , 21 , 22 ]. As we mentioned later, previous studies have sporadically reported some data quality elements in health information systems. There is no comprehensive agreement on its dimensions and there is no unique accepted definition of data quality among researchers for health information systems. However, there is still a lack of a review compiling and synthesizing all elements introduced in the literature. In this study, a more comprehensive understanding of the elements for quality of data in health information systems has been done using a systematic review method. The findings of this study can provide opportunities for health policy maker to become familiar with various data quality elements in health information. This systematic review specifically answered the following research questions:

1- What are the common data quality elements for health information systems?

2- What are the roles of common data quality elements to improve the performance of health information systems?

In this review, we used a systematic approach to retrieve the relevant research studies. Our reporting strategy follows the PRISMA guidelines [ 23 ].

Eligibility criteria

In this study the inclusion criteria were: (1) Data quality components were showcased within a health information system; (2) published from the year 2003 to 2024; (3) empirical studies that answered the research questions or tested the hypothesis and conducted on specific health system The exclusion criteria were: (1) Research that did not outline data quality dimensions in health management systems; (2) Content presented in a format other than a scientific article such as Conference papers, book sections, and …; (4) Methodologies deemed to be deficient in terms of quality; (5) Publication language not in English; and (7) The full text was unavailable.

Information sources

The literature search was conducted between September and October 2023, using the following five electronic scientific databases: Web of Knowledge, Science Direct, Emerald, PubMed, Scopus and Google Scholar search engine as an additional source for tracing references.

Search strategy

This study used a systematized review approach to identify common data quality elements for health information systems. The following keywords were used in the search strategy: Data quality, Health, clinic, Hospital, Medical, Information system. The keywords chosen were searched using various combinations and in the fields of title, abstract, subject, and keyword. We considered the search features in each database and used the Boolean operators (AND, OR) to combine and search selected keywords. An example of the search strategy was given in Table  1 .

Study selection

All the results were imported into EndNote reference management software. The duplicate and non-journal papers were removed. Next, the title and abstract of the remaining articles were screened to detect subject relevance with the research objectives. The selected articles were analyzed based on the inclusion and exclusion criteria. Finally, the reference lists of all identified articles were searched for additional studies. Two researchers undertook the screening of titles and abstracts obtained through the searches. A sample of just over 20% of articles was double screened in order to assess the level of agreement between the researchers. Disagreements were resolved through discussion or consultation with a third researcher.

Data collection process

Data extraction was completed independently by two assessors. The data were extracted from including four sections: bibliographic information, methodology, and the data quality elements investigated, and key findings. Each study was treated as a single unit of analysis and the relevant information in each study was extracted using a designated data extraction form.

Information was extracted from each included study (including first author, title, publication date, type of study, methodology, processes of knowledge management that were studied and selected results). We emphasize the results of selected papers that have reported elements for assessment data quality in health information systems.

Risk of bias in individual studies

In this study, we used the Joanna Briggs Institute (JBI) checklist [ 24 ] for quality assessment. The authors assessed the included studies with a further random examination by two independent reviewers. The results of the quality assessment were compared any disagreements between the reviewers were addressed through discussion or by involving a third reviewer.

Synthesis of results

In this review, by adopting similar identifies elements as broader themes, the results of the included studies were analyzed and categorized. Finally, the homogeneous data quality elements in health information systems were synthesized and described.

Risk of bias within studies

The JBI checklist was applied to all 58 studies; none were excluded based on quality assessment and all studies were rated as unclear or high risk of bias. In 16% of studies, we cannot find “statement locating the researcher culturally or theoretically” and in 37%, “influence of the researcher on the research” is not addressed.

The search for systematic reviews identified 734 references published between 2003 and 2024. Title and abstract review selected 167 references for full text review. In the analysis, it was found that 68 papers did not address research questions or test hypotheses, 32 papers lacked discussion on data quality dimensions in health management systems, and nine documents presented content in a format other than a scientific article.

Out of the 58 selected paper for final review, 42 were released between 2013 and 2024 [ 1 , 4 , 5 , 7 , 8 , 9 , 10 , 11 , 14 , 15 , 16 , 17 , 18 , 21 , 22 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 ]. Thirteen papers looked at information quality [ 7 , 11 , 14 , 27 , 28 , 29 , 31 , 37 , 52 , 54 , 55 , 56 ], five at content quality [ 7 , 15 , 21 , 43 , 50 ], and thirty-six at data quality [ 4 , 5 , 10 , 14 , 17 , 20 , 21 , 27 , 28 , 29 , 31 , 32 , 33 , 36 , 37 , 42 , 43 , 44 , 47 , 49 , 50 , 51 , 52 , 53 , 55 , 57 , 58 , 59 , 60 ]. None of the publications, however, made a distinction between “data” and “information,” or between “data quality” and “information quality.” As a result, “information quality” and “data quality” were used synonymously [ 21 ]. The search results and the study selection process are presented in Fig.  1 .

Flow diagram of study selection process

Evaluating the quality of the data was the primary goal of the reviewed studies [ 4 , 5 , 10 , 13 , 14 , 15 , 17 , 18 , 19 , 20 , 21 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 35 , 36 , 37 , 38 , 39 , 41 , 42 , 43 , 44 , 45 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 55 , 56 , 57 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 ].Two paper focused on information quality in health systems [ 11 , 52 ]. Methods for evaluating the quality of data were presented in eight publications [ 10 , 20 , 21 , 35 , 38 , 41 , 51 , 52 ], 19 publications tended to conduct on the health information [ 5 , 8 , 10 , 11 , 16 , 17 , 20 , 21 , 22 , 26 , 31 , 37 , 42 , 47 , 49 , 50 , 51 , 55 , 57 , 60 , 66 ] and eight paper focus on health or medical records as an information system in health context [ 13 , 19 , 25 , 38 , 44 , 45 , 64 , 67 ].

To describe data quality, the studies employed a total of 57 dimensions. The first data quality attribute for health information system that was most often used was accuracy [ 4 , 5 , 15 , 17 , 19 , 28 , 29 , 32 , 33 , 34 , 37 , 41 , 43 , 45 , 46 , 49 , 51 , 53 , 59 ], second is completeness [ 4 , 5 , 20 , 28 , 29 , 30 , 41 , 44 , 45 , 46 , 48 , 49 , 51 , 52 , 53 , 56 ], and third most-frequently criterion is timeliness [ 5 , 28 , 41 , 44 , 45 , 51 ]. Table  2 displays the common dimensions of data quality in health information systems that derived from existing literature.

Data accuracy measures the extent to which information accurately represents the objects or events. The accuracy of the information that is gathered, utilized, and stored is assessed through data accuracy. It is imperative for records to serve as a dependable source of information and to facilitate the generation of valuable insights through analysis. Maintaining high data accuracy guarantees that records and datasets meet the standards for reliability and trustworthiness, allowing for their use in decision-making and various applications [ 4 , 5 , 17 , 28 , 29 , 32 , 34 ]. Correctness, precision, free of error, validity, believability and integrity are common terms that use for describe data accuracy [ 21 ]. Data believability relates to whether the data is regarded as being true, real, and credible. Data believability is based on user’s perceptions [ 1 , 36 , 40 ].

Data consistency is the state in which all copies or instances of data are identical across various information systems. This uniformity is crucial in maintaining the accuracy, currency, and coherence of data across different platforms and applications. It is essential for instilling trust in users accessing the data. Implementing data validation rules, employing data standardization techniques, and utilizing data synchronization processes are some strategies to uphold data consistency. By ensuring data consistency, organizations can provide users with reliable information for making informed decisions, streamline operations, minimize errors, and enhance efficiency [ 9 , 45 , 48 , 51 , 52 , 65 ].

Data security is the practice of protecting information from corruption, theft, or unauthorized access throughout its life cycle. This involves safeguarding hardware, software, storage devices, and user devices, as well as implementing access controls, administrative controls, and organizational policies. By utilizing tools and technologies that enhance visibility of data usage, such as data masking, encryption, and redaction, organizations can ensure the security of their data. Moreover, data security assists organizations in streamlining auditing procedures and complying with data protection regulations, ultimately reducing the risk of cyber-attacks, human error, and insider threats [ 5 , 48 , 56 ]. Secure access, safe, confidentiality and privacy are common terms that use for describe data security [ 21 ].

Data timeliness denotes the currency and availability of data at the required time for its intended use. This is critical for enabling health organizations to make swift and accurate decisions based on the most up-to-date information. The timeliness of data has an impact on data quality as it determines the reliability and usefulness of information systems. Moreover, timely data can lead to cost savings as organizations can utilize real-time data to effectively manage inventories, optimize delivery routes, and coordinate with suppliers, thus reducing the risk of stock outs, minimizing delivery delays, and ensuring smooth operations [ 5 , 25 , 28 , 41 , 44 , 45 , 51 ].

Completeness of data refers to the extent to which information includes all necessary elements and observations for a specific purpose. This factor enhances the integrity and reliability of analyses, preventing gaps in understanding and supporting more robust decision-making processes. In a complete dataset, all variables relevant to the presentation of information should be present and fully populated with valid data values. Any missing, incorrect, or incomplete entries in the dataset can compromise the quality of analyses, interpretations, and decisions based on that data [ 4 , 5 , 9 , 28 , 29 , 30 , 41 , 44 , 45 , 52 ]. Coverage, comprehensiveness, appropriate amount, adequate, appropriate amount of data and integrity are common terms that use for describe data completeness [ 21 ]. The amount of data indicates the extent of data sets obtained for analysis and processing. In present-day information systems, these sets of data are frequently observed to be escalating in size, reaching capacities such as terabytes and petabytes [ 4 , 29 , 50 , 57 ].

Data reliability pertains to the uniformity of data across various records, programs, or platforms, as well as the credibility of the data source. Reliable data remains consistently accurate, while unreliable data may not always be valid, making it challenging to ascertain its accuracy. Consequently, organizations cannot depend on unreliable data for decision-making. Data reliability, also referred to as data observability, represents the trustworthiness of data and the insights derived from it for enabling sound decision-making. Reliability is characterized by two other fundamental elements of data quality include accuracy and consistency [ 9 , 49 , 53 , 57 , 59 , 65 ].

Data accessibility refers to the ease with which users can locate, retrieve, comprehend, and utilize data within an organization’s information systems. This is crucial in the modern digital landscape, where data is valuable for decision-making, strategic planning, and operational efficiency. Ensuring data accessibility involves creating an environment where data is available, understandable, and usable by individuals with varying levels of technical expertise. This approach is closely tied to data democratization, which aims to break down silos and make data available across different levels and departments of an organization. A well-implemented data accessibility strategy ensures that data is not locked away in isolated information systems but is integrated and accessible, contributing to a more informed and agile organizational structure. The ultimate goal is to empower users to leverage data in their daily tasks and decision-making processes, thus fostering a data-driven culture [ 4 , 26 , 29 , 33 , 50 , 57 ].

Data Objectivity refers to the extent to which data is free from personal biases, emotions, and subjective interpretations. Objective data is verifiable, reliable, and accurate, meaning that it can be verified independently by multiple parties. In other words, objective data is based on facts rather than opinions or judgments. In the context of information systems, data objectivity is crucial because it enables organizations to make informed decisions based on accurate and reliable information. Objective data helps to reduce errors, inconsistencies, and uncertainties, ensuring that business processes are efficient, effective, and compliant with regulatory requirements. Data objectivity in information systems is often hindered by biases in data collection, data quality issues, information overload, and lack of standardization. Biases may arise from human error, sampling errors, or deliberate data manipulation during the collection process. Inaccuracies, inconsistencies, and incompleteness resulting from poor data quality can compromise the objectivity of the information. The overwhelming amount of data available can make it challenging to differentiate between objective and subjective information. Inconsistencies in data representation and interpretation may occur due to the use of different systems or formats [ 36 , 41 , 44 , 45 , 46 ].

Data relevancy is an aspect of data quality that determines whether the data used or generated are relevant to add to the new target system and how usable it is for users [ 9 , 29 , 45 , 48 , 51 ]. Ease of operation, Usability, applicable, utility, Usefulness, Perceived usefulness and importance are common terms that use for describe data relevancy [ 21 ]. The concept of data usability revolves around a user’s ability to obtain meaningful information from various systems. When data is stored in text files that demand prolonged and intricate processing before it can be analyzed, its usability is limited. Conversely, data that is conveniently displayed on a performance dashboard for immediate interpretation is classified as highly usable [ 4 , 25 , 29 , 45 , 48 , 50 ]. The concept of data usefulness denotes the level at which data, post-analysis, aligns with the intended purpose within a given context for its user or consumer. In most cases, data usefulness is attained when all criteria related to data quality, such as dependability, thoroughness, uniformity, and others, are fulfilled [ 43 , 50 , 52 ].

Data Understandability refer to the level at which data exhibits qualities that facilitate understanding and analysis by users, and are presented in relevant languages, symbols, and measurements within a defined context of utilization [ 22 , 34 , 37 , 46 ]. Interpretability, ease of understanding, granularity and transparency are common terms that use for describe data understandability [ 21 ].

Data navigation refers to the process of searching, locating, and extracting relevant data from a vast pool of information to support decision-making, problem-solving, or analysis. It involves the utilization of different techniques and tools to navigate through extensive data, identify patterns, trends, and correlations, and present the information in a meaningful and actionable way. The success of data navigation is contingent upon several dimensions, including technical, domain knowledge, systems, methodological, and human dimensions. The technical dimension involves mastering programming languages like SQL and Python, utilizing data visualization software such as Tableau and Power BI, and implementing data mining techniques like machine learning algorithms. Domain knowledge dimension stresses the importance of expertise in specific fields. Information system dimension highlights the role of databases, data warehouses, cloud storage platforms, and other technologies in facilitating data navigation by storing, managing, and providing access to data. Methodological dimension focuses on statistical analysis, data mining techniques, and data visualization methods as key approaches to navigating data. Lastly, human dimension recognizes the significance of communication skills, collaboration, and critical thinking in the process of data navigation [ 4 , 50 , 65 , 68 ].

Data reputation is the evaluation of the trustworthiness, reliability, and credibility of data in an information system. It signifies the extent to which stakeholders, such as users, decision-makers, and other systems, perceive the data as accurate, reliable, and complete. Within an information system, data reputation plays a crucial role in decision-making, trust, system performance, and data sharing [ 42 , 60 , 61 ].

The concept of data efficiency revolves around an organization’s effectiveness in maximizing the value obtained from its data, while simultaneously minimizing the resources essential for processing, storing, and up keeping that data. Put simply, data efficiency focuses on streamlining the collection, storage, analysis, and utilization of data to meet objectives. When considering an information system, data efficiency can be examined from various angles, such as efficiency in data acquisition, storage, processing, analysis, visualization, security, retention, and archiving [ 7 , 28 , 29 , 48 ].

Data value-added pertains to the process of refining raw data into more useful, meaningful, and valuable information that can support decision-making, drive business outcomes, and create a competitive advantage. This process involves extracting insights, patterns, or trends from large datasets and presenting them in a manner that is easy to understand and act upon. By prioritizing these dimensions of data value-added within an information system, organizations can ensure that their data is transformed into valuable insights that support informed decision-making and drive business outcomes [ 5 , 22 , 25 , 45 ].

In a few papers, the concept of “fitness for use” was applied to data quality [ 6 , 55 , 69 ]. Two viewpoints can be used to characterize data quality: (1) the inherent quality of the data elements and set, and (2) how the set satisfies the needs of the user. The definition provided by the International Standards Organization best captures the accepted meaning of data quality, which is “the totality of features and characteristics of an entity that bears on its ability to satisfy stated and implied needs” [ 4 , 15 , 28 , 33 , 53 ].

Current review study identified 14 common dimensions for data quality in health information system. In related research data quality dimensions classified on four dimensions include: intrinsic (accuracy, objectivity, reputation), contextual timeliness, completeness, and relevancy), representational (representational format, understandability, consistency), and accessibility (accessibility, security) categories [ 53 , 60 , 69 , 70 , 71 ]. There exists a certain level of intersection between the aspects of data quality recognized in this review and those research in prior classifications of data quality.

Previous literature has often discussed intrinsic data quality in terms of the absence of defects, as indicated by various dimensions such as accuracy, perfection, freshness, and uniformity [ 72 ]. and “completeness, unambiguity, meaningless and correctness” [ 54 , 73 , 74 ]. The Canadian Institute for Health Information put forth a set of 69 quality criteria, organized into 24 quality characteristics, and further classified into 6 quality dimensions: accuracy, timeliness, comparability, usability, relevance, and privacy & security [ 58 , 71 ]. Research on data quality has primarily concentrated on recognizing general quality traits like accuracy, currency, completeness, correctness, consistency, and timeliness as fundamental aspects of data quality applicable across different fields. Nevertheless, existing reviews reveal a lack of consensus regarding the conceptual framework and definition of data quality [ 70 , 73 ]. However, our pervious review shows there is a lack of consensus conceptual framework and definition for data quality [ 1 , 71 ].

In this study, the three most-frequently used dimensions of data quality were accuracy, completeness and timeliness, respectively. This arrangement is somewhat different from previous literature in which the three most-frequently used dimensions were arranged in the order of completeness, accuracy, and timeliness, respectively [ 43 , 51 , 53 ]. Furthermore, the absence of a precise definition of the data quality dimensions led to complexities in evaluating them. The definitions of dimensions and their associated metrics were occasionally based on intuition, past experiences, or the underlying goals. These results indicate that data quality is a multi-faceted phenomenon. Likewise, other scholars argue that data quality is a multi-dimensional notion [ 5 , 28 , 38 , 52 , 61 ].

The Health Information Systems heavily rely on data, as they perform essential functions like generation, compilation, analysis, synthesis, communication, and data application to support decision-making. The literature frequently evaluates the dimensions of data quality, but there is currently a lack of consistency and potential generalizability in using these dimensions and methods to assess data quality in Health Information Systems. In this review of the literature, the data quality for health information system were examined and identified 14 common dimension include: Accuracy, Consistency, Security, Timeliness, Completeness, Reliability, Accessibility, Objectivity, Relevancy, Understandability, Navigation, Reputation, Efficiency and Value- added.

The quality of data in health information systems is indispensable for healthcare institutions to make well-informed decisions and provide patients with optimal care. Accurate and timely data assists healthcare organizations and professionals in identifying patterns, predicting outcomes, and enhancing patient results. Conversely, inadequate data quality in healthcare or other data-related issues can lead to inaccurate diagnoses, inappropriate treatments, and harm to patients. To ensure data quality in healthcare, organizations must prioritize investments in data governance, data management, and data analysis tools, while also maintaining a continuous process of monitoring and improving data quality in health information systems.

It is essential to have high-quality data in order to ensure the safe and dependable delivery of healthcare services. Health facility data plays a crucial role in monitoring performance. While various organizations may prioritize different aspects of data quality, it is important to acknowledge that no health data, regardless of its source, can be deemed flawless. All data are susceptible to various limitations related to data quality, including missing values, bias, measurement error, and human errors in data entry and computation. These limitations are associated with technical, behavioral, and organizational factors [ 75 ].

This study has limitations. Firstly, the number of articles with complete data was relatively small. Secondly, assessing the quality of some studies were difficult because the quality assessment criteria were not clearly identified. We have proposed four fundamental implications to inspire future research. Firstly, it is crucial for researchers to give equal attention to all dimensions of data quality, as these dimensions can have both direct and indirect effects on data quality outcomes. Secondly, researchers should aim to evaluate the existing data quality models and frameworks through a combination of mixed methods and case study designs. Thirdly, it is important to identify the underlying causes of data quality issues in health information systems. Lastly, efforts should be made to develop interventions that can effectively address and prevent data quality issues from occurring.

Data availability

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Joanna Briggs Institute

Liaw S-T, et al. Quality assessment of real-world data repositories across the data life cycle: a literature review. J Am Med Inform Assoc. 2021;28(7):1591–9.

Article   PubMed   PubMed Central   Google Scholar  

WHO. Data Quality Assurance (DQA) . Health Service Data 2022 [cited 2022 2022]; https://www.who.int/data/data-collection-tools/health-service-data/data-quality-assurance-dqa#:~:text=WHO%20has%20produced%20the%20Data,annual%20data%20quality%20desk%20review

FMoH E. Health sector transformation plan . 2015, Addis Ababa, Ethiopia.

Rumisha SF, et al. Data quality of the routine health management information system at the primary healthcare facility and district levels in Tanzania. BMC Med Inf Decis Mak. 2020;20(1):340.

Article   Google Scholar  

Chekol A, et al. Data quality and associated factors of routine health information system among health centers of West Gojjam Zone, northwest Ethiopia, 2021. Front Health Serv. 2023;3:1059611.

Pipino LL, Lee YW, Wang RY. Data quality assessment. Commun ACM. 2002;45(4):211–8.

Ouedraogo M, et al. A quality assessment of Health Management Information System (HMIS) data for maternal and child health in Jimma Zone, Ethiopia. PLoS ONE. 2019;14(3):e0213600.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Lemma S, et al. Improving quality and use of routine health information system data in low-and middle-income countries: a scoping review. PLoS ONE. 2020;15(10):e0239683.

Bammidi TR, et al. The crucial role of Data Quality in Automated decision-making systems. Int J Manage Educ Sustainable Dev. 2024;7(7):22.

Google Scholar  

Adane A, et al. Exploring data quality and use of the routine health information system in Ethiopia: a mixed-methods study. BMJ open. 2021;11(12):e050356.

Mohammed SA, Yusof MM. Towards an evaluation framework for information quality management (IQM) practices for health information systems–evaluation criteria for effective IQM practices. J Eval Clin Pract. 2013;19(2):379–87.

Article   PubMed   Google Scholar  

Long J, Seko C. A New Method for Database Data Quality Evaluation at the Canadian Institute for Health Information (CIHI) . in ICIQ . 2002. Citeseer.

Adeleke IT, et al. Data quality assessment in healthcare: a 365-day chart review of inpatients’ health records at a Nigerian tertiary hospital. J Am Med Inform Assoc. 2012;19(6):1039–42.

Singh M, et al. Health management information system data quality under NRHM in District Sonipat, Haryana. Int J Health Sci Res (IJHSR). 2016;6(9):11–4.

CAS   Google Scholar  

Harrison K, Rahimi N. Carolina Danovaro-Holliday, factors limiting data quality in the expanded programme on immunization in low and middle-income countries: a scoping review . Vaccine. 2020;38(30):4652–63.

Shama AT, et al. Assessment of quality of routine health information system data and associated factors among departments in public health facilities of Harari region, Ethiopia. BMC Med Inf Decis Mak. 2021;21(1):1–12.

Bosch-Capblanch X, et al. Does an innovative paper-based health information system (PHISICC) improve data quality and use in primary healthcare? Protocol of a multicountry, cluster randomised controlled trial in sub-saharan African rural settings. BMJ Open. 2021;11(7):e051823.

Ehsani-Moghaddam B, Martin K, Queenan JA. Data quality in healthcare: a report of practical experience with the Canadian Primary Care Sentinel Surveillance Network data. Health Inform Manage J. 2021;50(1–2):88–92.

Brown PJB, Warmington V. Data quality probes—exploiting and improving the quality of electronic patient record data and patient care. Int J Med Informatics. 2002;68(1):91–8.

Lima CR, et al. [Review of data quality dimensions and applied methods in the evaluation of health information systems]. Cad Saude Publica. 2009;25(10):2095–109.

Alipour J, Ahmadi M. Dimensions and assessment methods of data quality in health information systems. Acta Med Mediterranea. 2017;33(2):313–20.

Tolera A et al. Barriers to healthcare data quality and recommendations in public health facilities in dire Dawa city administration, eastern Ethiopia: a qualitative study. Front Digit Health, 2024. 6.

Vrabel M. M. Preferred reporting items for systematic reviews and meta-analyses . In Oncology nursing forum . Oncology Nursing Society; 2015.

JBI QARI Critical appraisal checklist for interpretive & critical research . The Joanna Briggs Institute, Adelaide 2018; http://joannabriggs.org/research/critical-appraisal-tools.html

Fraser HSF, et al. Factors Influencing Data Quality in Electronic Health Record Systems in 50 Health Facilities in Rwanda and the role of clinical Alerts: cross-sectional observational study. JMIR Public Health Surveill. 2024;10:e49127.

Madandola OO, et al. The relationship between electronic health records user interface features and data quality of patient clinical information: an integrative review. J Am Med Inform Assoc. 2023;31(1):240–55.

Getachew N, Erkalo B, Garedew MG. Data quality and associated factors in the health management information system at health centers in Shashogo district, Hadiya Zone, southern Ethiopia, 2021. Volume 22. BMC Medical Informatics and Decision Making; 2022. pp. 1–9. 1.

Solomon M, et al. Data quality assessment and associated factors in the health management information system among health centers of Southern Ethiopia. PLoS ONE. 2021;16(10):e0255949.

Moukénet A, et al. Health management information system (HMIS) data quality and associated factors in Massaguet district, Chad. BMC Med Inf Decis Mak. 2021;21(1):326.

do Einloft N. Data quality and arbovirus infection associated factors in pregnant and non-pregnant women of childbearing age in Brazil: a surveillance database analysis. One Health. 2021;12:100244.

Ayele W et al. Data quality and it’s correlation with routine health information system structure and input at public health centers in Addis Ababa, Ethiopia. Ethiop J Health Dev, 2021. 35(1).

Mulissa Z, et al. Effect of data quality improvement intervention on health management information system data accuracy: an interrupted time series analysis. PLoS ONE. 2020;15(8):e0237703.

Yourkavitch J, Prosnitz D, Herrera S. Data quality assessments stimulate improvements to health management information systems: evidence from five African countries. J Glob Health. 2019;9(1):010806.

Endriyas M, et al. Understanding performance data: health management information system data accuracy in Southern Nations nationalities and people’s Region, Ethiopia. BMC Health Serv Res. 2019;19(1):1–6.

Biancone P, et al. Data quality methods and applications in health care system: a systematic literature review. Int J Bus Manage. 2019;14(4):35–47.

Liu Y, et al. [Designing and implementation of the data quality control in the information system of air pollution and health impact monitoring]. Wei Sheng Yan Jiu. 2018;47(2):277–80.

PubMed   Google Scholar  

Kumar M, et al. Research gaps in routine health information system design barriers to data quality and use in low- and middle-income countries: a literature review. Int J Health Plann Manage. 2018;33(1):e1–9.

Feder SL. Data quality in electronic health records research: quality domains and assessment methods. West J Nurs Res. 2018;40(5):753–66.

Watson NL, et al. Data management and data quality in PERCH, a large international case-control study of severe childhood pneumonia. Clin Infect Dis. 2017;64(suppl3):S238–44.

Wagenaar BH, et al. Data-driven quality improvement in low-and middle-income country health systems: lessons from seven years of implementation experience across Mozambique, Rwanda, and Zambia. BMC Health Serv Res. 2017;17:65–75.

Puttkammer N, et al. Identifying priorities for data quality improvement within Haiti׳s iSanté EMR system: comparing two methods. Health Policy Technol. 2017;6(1):93–104.

Finnegan K, et al. Barriers and facilitators of Data Quality and Use in Malawi’s Health Information System. Annals Global Health. 2017;83(1):36–7.

Chen H, et al. Data Quality of the Chinese National AIDS Information System: a critical review. Stud Health Technol Inf. 2017;245:1352.

Woinarowicz M, Howell M. The impact of electronic health record (EHR) interoperability on immunization information system (IIS) data quality. Online J Public Health Inf. 2016;8(2):e184.

Puttkammer N, et al. An assessment of data quality in a multi-site electronic medical record system in Haiti. Int J Med Informatics. 2016;86:104–16.

Article   CAS   Google Scholar  

Nicol E, Dudley L, Bradshaw D. Assessing the quality of routine data for the prevention of mother-to-child transmission of HIV: an analytical observational study in two health districts with high HIV prevalence in South Africa. Int J Med Informatics. 2016;95:60–70.

Wagenaar BH, et al. Effects of a health information system data quality intervention on concordance in Mozambique: time-series analyses from 2009–2012. Popul Health Metr. 2015;13:9.

Taggart J, Liaw S-T, Yu H. Structured data quality reports to improve EHR data quality. Int J Med Informatics. 2015;84(12):1094–8.

Glèlè Ahanhanzo Y, et al. Data quality assessment in the routine health information system: an application of the Lot Quality Assurance Sampling in Benin. Health Policy Plan. 2015;30(7):837–43.

Glèlè Ahanhanzo Y, et al. Factors associated with data quality in the routine health information system of Benin. Arch Public Health. 2014;72(1):25.

Chen H, et al. A review of data quality assessment methods for public health information systems. Int J Environ Res Public Health. 2014;11(5):5170–207.

Hahn D, Wanjala P, Marx M. Where is information quality lost at clinical level? A mixed-method study on information systems and data quality in three urban Kenyan ANC clinics. Glob Health Action. 2013;6:21424.

Chen H, Yu P, Wang N. Do we have the reliable data? An exploration of data quality for AIDS information system in China. Stud Health Technol Inf. 2013;192:1042.

Choquet R, et al. The Information Quality Triangle: a methodology to assess clinical information quality , in MEDINFO 2010 . IOS; 2010. pp. 699–703.

Mettler T, Rohner P, Baacke L. Improving data quality of health information systems: a holistic design-oriented approach. 2008.

Sørensen HT, et al. Identification of cases of meningococcal disease: data quality in two Danish population-based information systems during a 14-year period. Int J Risk Saf Med. 1995;7(3):179–89.

Gimbel S, et al. An assessment of routine primary care health information system data quality in Sofala Province, Mozambique. Popul Health Metr. 2011;9:12.

Kerr K, Norris T, Stockdale R. Data quality information and decision making: a healthcare case study. ACIS 2007 proceedings, 2007: p. 98.

Ben Saïd M, et al. A multi-source information System via the internet for end-stage renal disease: Scalability and Data Quality. Stud Health Technol Inf. 2005;116:994–9.

Fletcher DM. Achieving data quality. How data from a pediatric health information system earns the trust of its users. J Ahima. 2004;75(10):22–6.

Bean KP. Data quality in hospital strategic information systems: a summary of survey findings. Top Health Inf Manage. 1994;15(2):13–25.

CAS   PubMed   Google Scholar  

Kelly A, Becker W. Nutrition information systems and data quality requirements. WHO Reg Publ Eur Ser. 1991;34:15–24.

Leitheiser RL. Data quality in health care data warehouse environments . in Proceedings of the 34th annual Hawaii international conference on system sciences . 2001. IEEE.

Ndira S, Rosenberger K, Wetter T. Assessment of data quality of and staff satisfaction with an electronic health record system in a developing country (Uganda). Methods Inf Med. 2008;47(06):489–98.

Article   CAS   PubMed   Google Scholar  

Silva AA, et al. [Evaluation of data quality from the information system on live births in 1997–1998]. Rev Saude Publica. 2001;35(6):508–14.

Woelk GB, Moyo IM, Ray CS. A health information system revised. Part II: improving data quality and utilization. Cent Afr J Med. 1987;33(7):170–3.

Abbasi R, Khajouei R, Sadeqi M, Jabali. Timeliness and accuracy of information sharing from hospital information systems to electronic health record in Iran. J Health Adm. 2019;22(2):28–40.

Elavsky F, Nadolskis L, Moritz D. Data navigator: an accessibility-centered data navigation toolkit. IEEE Trans Vis Comput Graph. 2023;20(1):16–25.

Wang RY. A product perspective on total data quality management. Commun ACM. 1998;41(2):58–65.

Liaw S-T et al. Data quality and fitness for purpose of routinely collected data–a general practice case study from an electronic practice-based research network (ePBRN) . in AMIA Annual Symposium Proceedings . 2011. American Medical Informatics Association.

Rahimi A, et al. Ontological specification of quality of chronic disease data in EHRs to support decision analytics: a realist review. Decis Analytics. 2014;1:1–31.

Redman TC. Measuring data accuracy: A framework and review. Information quality, 2014: pp. 21–36.

Orme AM, Yao H, Etzkorn LH. Indicating ontology data quality, stability, and completeness throughout ontology evolution. J Softw Maintenance Evolution: Res Pract. 2007;19(1):49–75.

Yao H, Orme AM, Etzkorn L. Cohesion metrics for ontology design and application. J Comput Sci. 2005;1(1):107–13.

Endriyas M, et al. Understanding performance data: health management information system data accuracy in Southern Nations nationalities and people’s Region, Ethiopia. BMC Health Serv Res. 2019;19(175):1–6.

Download references

Acknowledgements

Not applicable.

This study was supported by Abadan University of medical sciences, Research code: 1557.

Author information

Authors and affiliations.

Department of Medical library and Information Science, Abadan University of Medical Sciences, Abadan, Iran

Hossein Ghalavand & Zarrin Zarrinabadi

Department of Medical library and Information Science, School of Health Management and Information Sciences, Isfahan University of Medical Sciences, Isfahan, Iran

Saied Shirshahi & Alireza Rahimi

Student Research Committee, Abadan University of Medical Sciences, Abadan, Iran

Fatemeh Amani

You can also search for this author in PubMed   Google Scholar

Contributions

Hossein Ghalavand and Saied Shirshahi Conceived the study, prepared the analysis plan, conducted the analysis, and prepared the draft manuscript. Alireza Rahimi, Zarrin Zarrinabadi and Fatemeh Amani Conceived the study, prepared the analysis plan, performed the literature search, screening for study inclusion/exclusion, and risk of bias assessment, conducted the analysis, and prepared the draft manuscript. All authors contributed to the final version of the manuscript.

Corresponding author

Correspondence to Hossein Ghalavand .

Ethics declarations

Ethics approval and consent to participate.

This research study was approved by Ethics Committee in Biomedical Research at Abadan University of Medical Sciences (Ethical code: IR.ABADANUMS.REC.1401.122).

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Ghalavand, H., Shirshahi, S., Rahimi, A. et al. Common data quality elements for health information systems: a systematic review. BMC Med Inform Decis Mak 24 , 243 (2024). https://doi.org/10.1186/s12911-024-02644-7

Download citation

Received : 17 April 2024

Accepted : 21 August 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s12911-024-02644-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data quality
• Health Information System
• Systematic review

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]