validity in research qualitative

Criteria for Good Qualitative Research: A Comprehensive Review

• Regular Article
• Open access
• Published: 18 September 2021
• Volume 31 , pages 679–689, ( 2022 )

Cite this article

You have full access to this open access article

• Drishti Yadav   ORCID: orcid.org/0000-0002-2974-0323 1  

100k Accesses

47 Citations

70 Altmetric

Explore all metrics

This review aims to synthesize a published set of evaluative criteria for good qualitative research. The aim is to shed light on existing standards for assessing the rigor of qualitative research encompassing a range of epistemological and ontological standpoints. Using a systematic search strategy, published journal articles that deliberate criteria for rigorous research were identified. Then, references of relevant articles were surveyed to find noteworthy, distinct, and well-defined pointers to good qualitative research. This review presents an investigative assessment of the pivotal features in qualitative research that can permit the readers to pass judgment on its quality and to condemn it as good research when objectively and adequately utilized. Overall, this review underlines the crux of qualitative research and accentuates the necessity to evaluate such research by the very tenets of its being. It also offers some prospects and recommendations to improve the quality of qualitative research. Based on the findings of this review, it is concluded that quality criteria are the aftereffect of socio-institutional procedures and existing paradigmatic conducts. Owing to the paradigmatic diversity of qualitative research, a single and specific set of quality criteria is neither feasible nor anticipated. Since qualitative research is not a cohesive discipline, researchers need to educate and familiarize themselves with applicable norms and decisive factors to evaluate qualitative research from within its theoretical and methodological framework of origin.

Similar content being viewed by others

Good Qualitative Research: Opening up the Debate

Beyond qualitative/quantitative structuralism: the positivist qualitative research and the paradigmatic disclaimer.

What is Qualitative in Research

Avoid common mistakes on your manuscript.

Introduction

“… It is important to regularly dialogue about what makes for good qualitative research” (Tracy, 2010 , p. 837)

To decide what represents good qualitative research is highly debatable. There are numerous methods that are contained within qualitative research and that are established on diverse philosophical perspectives. Bryman et al., ( 2008 , p. 262) suggest that “It is widely assumed that whereas quality criteria for quantitative research are well‐known and widely agreed, this is not the case for qualitative research.” Hence, the question “how to evaluate the quality of qualitative research” has been continuously debated. There are many areas of science and technology wherein these debates on the assessment of qualitative research have taken place. Examples include various areas of psychology: general psychology (Madill et al., 2000 ); counseling psychology (Morrow, 2005 ); and clinical psychology (Barker & Pistrang, 2005 ), and other disciplines of social sciences: social policy (Bryman et al., 2008 ); health research (Sparkes, 2001 ); business and management research (Johnson et al., 2006 ); information systems (Klein & Myers, 1999 ); and environmental studies (Reid & Gough, 2000 ). In the literature, these debates are enthused by the impression that the blanket application of criteria for good qualitative research developed around the positivist paradigm is improper. Such debates are based on the wide range of philosophical backgrounds within which qualitative research is conducted (e.g., Sandberg, 2000 ; Schwandt, 1996 ). The existence of methodological diversity led to the formulation of different sets of criteria applicable to qualitative research.

Among qualitative researchers, the dilemma of governing the measures to assess the quality of research is not a new phenomenon, especially when the virtuous triad of objectivity, reliability, and validity (Spencer et al., 2004 ) are not adequate. Occasionally, the criteria of quantitative research are used to evaluate qualitative research (Cohen & Crabtree, 2008 ; Lather, 2004 ). Indeed, Howe ( 2004 ) claims that the prevailing paradigm in educational research is scientifically based experimental research. Hypotheses and conjectures about the preeminence of quantitative research can weaken the worth and usefulness of qualitative research by neglecting the prominence of harmonizing match for purpose on research paradigm, the epistemological stance of the researcher, and the choice of methodology. Researchers have been reprimanded concerning this in “paradigmatic controversies, contradictions, and emerging confluences” (Lincoln & Guba, 2000 ).

In general, qualitative research tends to come from a very different paradigmatic stance and intrinsically demands distinctive and out-of-the-ordinary criteria for evaluating good research and varieties of research contributions that can be made. This review attempts to present a series of evaluative criteria for qualitative researchers, arguing that their choice of criteria needs to be compatible with the unique nature of the research in question (its methodology, aims, and assumptions). This review aims to assist researchers in identifying some of the indispensable features or markers of high-quality qualitative research. In a nutshell, the purpose of this systematic literature review is to analyze the existing knowledge on high-quality qualitative research and to verify the existence of research studies dealing with the critical assessment of qualitative research based on the concept of diverse paradigmatic stances. Contrary to the existing reviews, this review also suggests some critical directions to follow to improve the quality of qualitative research in different epistemological and ontological perspectives. This review is also intended to provide guidelines for the acceleration of future developments and dialogues among qualitative researchers in the context of assessing the qualitative research.

The rest of this review article is structured in the following fashion: Sect.  Methods describes the method followed for performing this review. Section Criteria for Evaluating Qualitative Studies provides a comprehensive description of the criteria for evaluating qualitative studies. This section is followed by a summary of the strategies to improve the quality of qualitative research in Sect.  Improving Quality: Strategies . Section  How to Assess the Quality of the Research Findings? provides details on how to assess the quality of the research findings. After that, some of the quality checklists (as tools to evaluate quality) are discussed in Sect.  Quality Checklists: Tools for Assessing the Quality . At last, the review ends with the concluding remarks presented in Sect.  Conclusions, Future Directions and Outlook . Some prospects in qualitative research for enhancing its quality and usefulness in the social and techno-scientific research community are also presented in Sect.  Conclusions, Future Directions and Outlook .

For this review, a comprehensive literature search was performed from many databases using generic search terms such as Qualitative Research , Criteria , etc . The following databases were chosen for the literature search based on the high number of results: IEEE Explore, ScienceDirect, PubMed, Google Scholar, and Web of Science. The following keywords (and their combinations using Boolean connectives OR/AND) were adopted for the literature search: qualitative research, criteria, quality, assessment, and validity. The synonyms for these keywords were collected and arranged in a logical structure (see Table 1 ). All publications in journals and conference proceedings later than 1950 till 2021 were considered for the search. Other articles extracted from the references of the papers identified in the electronic search were also included. A large number of publications on qualitative research were retrieved during the initial screening. Hence, to include the searches with the main focus on criteria for good qualitative research, an inclusion criterion was utilized in the search string.

From the selected databases, the search retrieved a total of 765 publications. Then, the duplicate records were removed. After that, based on the title and abstract, the remaining 426 publications were screened for their relevance by using the following inclusion and exclusion criteria (see Table 2 ). Publications focusing on evaluation criteria for good qualitative research were included, whereas those works which delivered theoretical concepts on qualitative research were excluded. Based on the screening and eligibility, 45 research articles were identified that offered explicit criteria for evaluating the quality of qualitative research and were found to be relevant to this review.

Figure  1 illustrates the complete review process in the form of PRISMA flow diagram. PRISMA, i.e., “preferred reporting items for systematic reviews and meta-analyses” is employed in systematic reviews to refine the quality of reporting.

PRISMA flow diagram illustrating the search and inclusion process. N represents the number of records

Criteria for Evaluating Qualitative Studies

Fundamental criteria: general research quality.

Various researchers have put forward criteria for evaluating qualitative research, which have been summarized in Table 3 . Also, the criteria outlined in Table 4 effectively deliver the various approaches to evaluate and assess the quality of qualitative work. The entries in Table 4 are based on Tracy’s “Eight big‐tent criteria for excellent qualitative research” (Tracy, 2010 ). Tracy argues that high-quality qualitative work should formulate criteria focusing on the worthiness, relevance, timeliness, significance, morality, and practicality of the research topic, and the ethical stance of the research itself. Researchers have also suggested a series of questions as guiding principles to assess the quality of a qualitative study (Mays & Pope, 2020 ). Nassaji ( 2020 ) argues that good qualitative research should be robust, well informed, and thoroughly documented.

Qualitative Research: Interpretive Paradigms

All qualitative researchers follow highly abstract principles which bring together beliefs about ontology, epistemology, and methodology. These beliefs govern how the researcher perceives and acts. The net, which encompasses the researcher’s epistemological, ontological, and methodological premises, is referred to as a paradigm, or an interpretive structure, a “Basic set of beliefs that guides action” (Guba, 1990 ). Four major interpretive paradigms structure the qualitative research: positivist and postpositivist, constructivist interpretive, critical (Marxist, emancipatory), and feminist poststructural. The complexity of these four abstract paradigms increases at the level of concrete, specific interpretive communities. Table 5 presents these paradigms and their assumptions, including their criteria for evaluating research, and the typical form that an interpretive or theoretical statement assumes in each paradigm. Moreover, for evaluating qualitative research, quantitative conceptualizations of reliability and validity are proven to be incompatible (Horsburgh, 2003 ). In addition, a series of questions have been put forward in the literature to assist a reviewer (who is proficient in qualitative methods) for meticulous assessment and endorsement of qualitative research (Morse, 2003 ). Hammersley ( 2007 ) also suggests that guiding principles for qualitative research are advantageous, but methodological pluralism should not be simply acknowledged for all qualitative approaches. Seale ( 1999 ) also points out the significance of methodological cognizance in research studies.

Table 5 reflects that criteria for assessing the quality of qualitative research are the aftermath of socio-institutional practices and existing paradigmatic standpoints. Owing to the paradigmatic diversity of qualitative research, a single set of quality criteria is neither possible nor desirable. Hence, the researchers must be reflexive about the criteria they use in the various roles they play within their research community.

Improving Quality: Strategies

Another critical question is “How can the qualitative researchers ensure that the abovementioned quality criteria can be met?” Lincoln and Guba ( 1986 ) delineated several strategies to intensify each criteria of trustworthiness. Other researchers (Merriam & Tisdell, 2016 ; Shenton, 2004 ) also presented such strategies. A brief description of these strategies is shown in Table 6 .

It is worth mentioning that generalizability is also an integral part of qualitative research (Hays & McKibben, 2021 ). In general, the guiding principle pertaining to generalizability speaks about inducing and comprehending knowledge to synthesize interpretive components of an underlying context. Table 7 summarizes the main metasynthesis steps required to ascertain generalizability in qualitative research.

Figure  2 reflects the crucial components of a conceptual framework and their contribution to decisions regarding research design, implementation, and applications of results to future thinking, study, and practice (Johnson et al., 2020 ). The synergy and interrelationship of these components signifies their role to different stances of a qualitative research study.

Essential elements of a conceptual framework

In a nutshell, to assess the rationale of a study, its conceptual framework and research question(s), quality criteria must take account of the following: lucid context for the problem statement in the introduction; well-articulated research problems and questions; precise conceptual framework; distinct research purpose; and clear presentation and investigation of the paradigms. These criteria would expedite the quality of qualitative research.

How to Assess the Quality of the Research Findings?

The inclusion of quotes or similar research data enhances the confirmability in the write-up of the findings. The use of expressions (for instance, “80% of all respondents agreed that” or “only one of the interviewees mentioned that”) may also quantify qualitative findings (Stenfors et al., 2020 ). On the other hand, the persuasive reason for “why this may not help in intensifying the research” has also been provided (Monrouxe & Rees, 2020 ). Further, the Discussion and Conclusion sections of an article also prove robust markers of high-quality qualitative research, as elucidated in Table 8 .

Quality Checklists: Tools for Assessing the Quality

Numerous checklists are available to speed up the assessment of the quality of qualitative research. However, if used uncritically and recklessly concerning the research context, these checklists may be counterproductive. I recommend that such lists and guiding principles may assist in pinpointing the markers of high-quality qualitative research. However, considering enormous variations in the authors’ theoretical and philosophical contexts, I would emphasize that high dependability on such checklists may say little about whether the findings can be applied in your setting. A combination of such checklists might be appropriate for novice researchers. Some of these checklists are listed below:

The most commonly used framework is Consolidated Criteria for Reporting Qualitative Research (COREQ) (Tong et al., 2007 ). This framework is recommended by some journals to be followed by the authors during article submission.

Standards for Reporting Qualitative Research (SRQR) is another checklist that has been created particularly for medical education (O’Brien et al., 2014 ).

Also, Tracy ( 2010 ) and Critical Appraisal Skills Programme (CASP, 2021 ) offer criteria for qualitative research relevant across methods and approaches.

Further, researchers have also outlined different criteria as hallmarks of high-quality qualitative research. For instance, the “Road Trip Checklist” (Epp & Otnes, 2021 ) provides a quick reference to specific questions to address different elements of high-quality qualitative research.

Conclusions, Future Directions, and Outlook

This work presents a broad review of the criteria for good qualitative research. In addition, this article presents an exploratory analysis of the essential elements in qualitative research that can enable the readers of qualitative work to judge it as good research when objectively and adequately utilized. In this review, some of the essential markers that indicate high-quality qualitative research have been highlighted. I scope them narrowly to achieve rigor in qualitative research and note that they do not completely cover the broader considerations necessary for high-quality research. This review points out that a universal and versatile one-size-fits-all guideline for evaluating the quality of qualitative research does not exist. In other words, this review also emphasizes the non-existence of a set of common guidelines among qualitative researchers. In unison, this review reinforces that each qualitative approach should be treated uniquely on account of its own distinctive features for different epistemological and disciplinary positions. Owing to the sensitivity of the worth of qualitative research towards the specific context and the type of paradigmatic stance, researchers should themselves analyze what approaches can be and must be tailored to ensemble the distinct characteristics of the phenomenon under investigation. Although this article does not assert to put forward a magic bullet and to provide a one-stop solution for dealing with dilemmas about how, why, or whether to evaluate the “goodness” of qualitative research, it offers a platform to assist the researchers in improving their qualitative studies. This work provides an assembly of concerns to reflect on, a series of questions to ask, and multiple sets of criteria to look at, when attempting to determine the quality of qualitative research. Overall, this review underlines the crux of qualitative research and accentuates the need to evaluate such research by the very tenets of its being. Bringing together the vital arguments and delineating the requirements that good qualitative research should satisfy, this review strives to equip the researchers as well as reviewers to make well-versed judgment about the worth and significance of the qualitative research under scrutiny. In a nutshell, a comprehensive portrayal of the research process (from the context of research to the research objectives, research questions and design, speculative foundations, and from approaches of collecting data to analyzing the results, to deriving inferences) frequently proliferates the quality of a qualitative research.

Prospects : A Road Ahead for Qualitative Research

Irrefutably, qualitative research is a vivacious and evolving discipline wherein different epistemological and disciplinary positions have their own characteristics and importance. In addition, not surprisingly, owing to the sprouting and varied features of qualitative research, no consensus has been pulled off till date. Researchers have reflected various concerns and proposed several recommendations for editors and reviewers on conducting reviews of critical qualitative research (Levitt et al., 2021 ; McGinley et al., 2021 ). Following are some prospects and a few recommendations put forward towards the maturation of qualitative research and its quality evaluation:

In general, most of the manuscript and grant reviewers are not qualitative experts. Hence, it is more likely that they would prefer to adopt a broad set of criteria. However, researchers and reviewers need to keep in mind that it is inappropriate to utilize the same approaches and conducts among all qualitative research. Therefore, future work needs to focus on educating researchers and reviewers about the criteria to evaluate qualitative research from within the suitable theoretical and methodological context.

There is an urgent need to refurbish and augment critical assessment of some well-known and widely accepted tools (including checklists such as COREQ, SRQR) to interrogate their applicability on different aspects (along with their epistemological ramifications).

Efforts should be made towards creating more space for creativity, experimentation, and a dialogue between the diverse traditions of qualitative research. This would potentially help to avoid the enforcement of one's own set of quality criteria on the work carried out by others.

Moreover, journal reviewers need to be aware of various methodological practices and philosophical debates.

It is pivotal to highlight the expressions and considerations of qualitative researchers and bring them into a more open and transparent dialogue about assessing qualitative research in techno-scientific, academic, sociocultural, and political rooms.

Frequent debates on the use of evaluative criteria are required to solve some potentially resolved issues (including the applicability of a single set of criteria in multi-disciplinary aspects). Such debates would not only benefit the group of qualitative researchers themselves, but primarily assist in augmenting the well-being and vivacity of the entire discipline.

To conclude, I speculate that the criteria, and my perspective, may transfer to other methods, approaches, and contexts. I hope that they spark dialog and debate – about criteria for excellent qualitative research and the underpinnings of the discipline more broadly – and, therefore, help improve the quality of a qualitative study. Further, I anticipate that this review will assist the researchers to contemplate on the quality of their own research, to substantiate research design and help the reviewers to review qualitative research for journals. On a final note, I pinpoint the need to formulate a framework (encompassing the prerequisites of a qualitative study) by the cohesive efforts of qualitative researchers of different disciplines with different theoretic-paradigmatic origins. I believe that tailoring such a framework (of guiding principles) paves the way for qualitative researchers to consolidate the status of qualitative research in the wide-ranging open science debate. Dialogue on this issue across different approaches is crucial for the impending prospects of socio-techno-educational research.

Amin, M. E. K., Nørgaard, L. S., Cavaco, A. M., Witry, M. J., Hillman, L., Cernasev, A., & Desselle, S. P. (2020). Establishing trustworthiness and authenticity in qualitative pharmacy research. Research in Social and Administrative Pharmacy, 16 (10), 1472–1482.

Article   Google Scholar  

Barker, C., & Pistrang, N. (2005). Quality criteria under methodological pluralism: Implications for conducting and evaluating research. American Journal of Community Psychology, 35 (3–4), 201–212.

Bryman, A., Becker, S., & Sempik, J. (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11 (4), 261–276.

Caelli, K., Ray, L., & Mill, J. (2003). ‘Clear as mud’: Toward greater clarity in generic qualitative research. International Journal of Qualitative Methods, 2 (2), 1–13.

CASP (2021). CASP checklists. Retrieved May 2021 from https://casp-uk.net/casp-tools-checklists/

Cohen, D. J., & Crabtree, B. F. (2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. The Annals of Family Medicine, 6 (4), 331–339.

Denzin, N. K., & Lincoln, Y. S. (2005). Introduction: The discipline and practice of qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The sage handbook of qualitative research (pp. 1–32). Sage Publications Ltd.

Google Scholar  

Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38 (3), 215–229.

Epp, A. M., & Otnes, C. C. (2021). High-quality qualitative research: Getting into gear. Journal of Service Research . https://doi.org/10.1177/1094670520961445

Guba, E. G. (1990). The paradigm dialog. In Alternative paradigms conference, mar, 1989, Indiana u, school of education, San Francisco, ca, us . Sage Publications, Inc.

Hammersley, M. (2007). The issue of quality in qualitative research. International Journal of Research and Method in Education, 30 (3), 287–305.

Haven, T. L., Errington, T. M., Gleditsch, K. S., van Grootel, L., Jacobs, A. M., Kern, F. G., & Mokkink, L. B. (2020). Preregistering qualitative research: A Delphi study. International Journal of Qualitative Methods, 19 , 1609406920976417.

Hays, D. G., & McKibben, W. B. (2021). Promoting rigorous research: Generalizability and qualitative research. Journal of Counseling and Development, 99 (2), 178–188.

Horsburgh, D. (2003). Evaluation of qualitative research. Journal of Clinical Nursing, 12 (2), 307–312.

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10 (1), 42–46.

Johnson, J. L., Adkins, D., & Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. American Journal of Pharmaceutical Education, 84 (1), 7120.

Johnson, P., Buehring, A., Cassell, C., & Symon, G. (2006). Evaluating qualitative management research: Towards a contingent criteriology. International Journal of Management Reviews, 8 (3), 131–156.

Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 23 (1), 67–93.

Lather, P. (2004). This is your father’s paradigm: Government intrusion and the case of qualitative research in education. Qualitative Inquiry, 10 (1), 15–34.

Levitt, H. M., Morrill, Z., Collins, K. M., & Rizo, J. L. (2021). The methodological integrity of critical qualitative research: Principles to support design and research review. Journal of Counseling Psychology, 68 (3), 357.

Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation, 1986 (30), 73–84.

Lincoln, Y. S., & Guba, E. G. (2000). Paradigmatic controversies, contradictions and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 163–188). Sage Publications.

Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology, 91 (1), 1–20.

Mays, N., & Pope, C. (2020). Quality in qualitative research. Qualitative Research in Health Care . https://doi.org/10.1002/9781119410867.ch15

McGinley, S., Wei, W., Zhang, L., & Zheng, Y. (2021). The state of qualitative research in hospitality: A 5-year review 2014 to 2019. Cornell Hospitality Quarterly, 62 (1), 8–20.

Merriam, S., & Tisdell, E. (2016). Qualitative research: A guide to design and implementation. San Francisco, US.

Meyer, M., & Dykes, J. (2019). Criteria for rigor in visualization design study. IEEE Transactions on Visualization and Computer Graphics, 26 (1), 87–97.

Monrouxe, L. V., & Rees, C. E. (2020). When I say… quantification in qualitative research. Medical Education, 54 (3), 186–187.

Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52 (2), 250.

Morse, J. M. (2003). A review committee’s guide for evaluating qualitative proposals. Qualitative Health Research, 13 (6), 833–851.

Nassaji, H. (2020). Good qualitative research. Language Teaching Research, 24 (4), 427–431.

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89 (9), 1245–1251.

O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19 , 1609406919899220.

Reid, A., & Gough, S. (2000). Guidelines for reporting and evaluating qualitative research: What are the alternatives? Environmental Education Research, 6 (1), 59–91.

Rocco, T. S. (2010). Criteria for evaluating qualitative studies. Human Resource Development International . https://doi.org/10.1080/13678868.2010.501959

Sandberg, J. (2000). Understanding human competence at work: An interpretative approach. Academy of Management Journal, 43 (1), 9–25.

Schwandt, T. A. (1996). Farewell to criteriology. Qualitative Inquiry, 2 (1), 58–72.

Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5 (4), 465–478.

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22 (2), 63–75.

Sparkes, A. C. (2001). Myth 94: Qualitative health researchers will agree about validity. Qualitative Health Research, 11 (4), 538–552.

Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2004). Quality in qualitative evaluation: A framework for assessing research evidence.

Stenfors, T., Kajamaa, A., & Bennett, D. (2020). How to assess the quality of qualitative research. The Clinical Teacher, 17 (6), 596–599.

Taylor, E. W., Beck, J., & Ainsworth, E. (2001). Publishing qualitative adult education research: A peer review perspective. Studies in the Education of Adults, 33 (2), 163–179.

Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19 (6), 349–357.

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16 (10), 837–851.

Download references

Open access funding provided by TU Wien (TUW).

Author information

Authors and affiliations.

Faculty of Informatics, Technische Universität Wien, 1040, Vienna, Austria

Drishti Yadav

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Drishti Yadav .

Ethics declarations

Conflict of interest.

The author declares no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Yadav, D. Criteria for Good Qualitative Research: A Comprehensive Review. Asia-Pacific Edu Res 31 , 679–689 (2022). https://doi.org/10.1007/s40299-021-00619-0

Download citation

Accepted : 28 August 2021

Published : 18 September 2021

Issue Date : December 2022

DOI : https://doi.org/10.1007/s40299-021-00619-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Evaluative criteria
• Find a journal
• Publish with us
• Track your research

• Subscribe to journal Subscribe
• Get new issue alerts Get alerts

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

Rigor or Reliability and Validity in Qualitative Research: Perspectives, Strategies, Reconceptualization, and Recommendations

Cypress, Brigitte S. EdD, RN, CCRN

Brigitte S. Cypress, EdD, RN, CCRN , is an assistant professor of nursing, Lehman College and The Graduate Center, City University of New York.

The author has disclosed that she has no significant relationships with, or financial interest in, any commercial companies pertaining to this article.

Address correspondence and reprint requests to: Brigitte S. Cypress, EdD, RN, CCRN, Lehman College and The Graduate Center, City University of New York, PO Box 2205, Pocono Summit, PA 18346 ( [email protected] ).

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site ( www.dccnjournal.com ).

Issues are still raised even now in the 21st century by the persistent concern with achieving rigor in qualitative research. There is also a continuing debate about the analogous terms reliability and validity in naturalistic inquiries as opposed to quantitative investigations. This article presents the concept of rigor in qualitative research using a phenomenological study as an exemplar to further illustrate the process. Elaborating on epistemological and theoretical conceptualizations by Lincoln and Guba, strategies congruent with qualitative perspective for ensuring validity to establish the credibility of the study are described. A synthesis of the historical development of validity criteria evident in the literature during the years is explored. Recommendations are made for use of the term rigor instead of trustworthiness and the reconceptualization and renewed use of the concept of reliability and validity in qualitative research, that strategies for ensuring rigor must be built into the qualitative research process rather than evaluated only after the inquiry, and that qualitative researchers and students alike must be proactive and take responsibility in ensuring the rigor of a research study. The insights garnered here will move novice researchers and doctoral students to a better conceptual grasp of the complexity of reliability and validity and its ramifications for qualitative inquiry.

Conducting a naturalistic inquiry in general is not an easy task. Qualitative studies are more complex in many ways than a traditional investigation. Quantitative research follows a structured, rigid, preset design with the methods all prescribed. In naturalistic inquiries, planning and implementation are simultaneous, and the research design can change or is emergent. Preliminary steps must be accomplished before the design is fully implemented from making initial contact and gaining entry to site, negotiating consent, building and maintaining trust, and identifying participants. The steps of a qualitative inquiry are also repeated multiple times during the process. As the design unfolds, the elements of this design are put into place, and the inquirer has minimal control and should be flexible. There is continuous reassessment and reiteration. Data collection is carried out using multiple techniques, and whatever the source maybe, it is the researcher who is the sole instrument of the study and the primary mode of collecting the information. All the while during these processes, the qualitative inquirer must be concerned with rigor. 1 Appropriate activities must be conducted to ensure that rigor had been attended to in the research process rather than only adhering to set criteria for rigor after the completion of the study. 1-4

Reliability and validity are 2 key aspects of all research. Researchers assert that rigor of qualitative research equates to the concepts reliability and validity and all are necessary components of quality. 5,6 However, the precise definition of quality has created debates among naturalistic inquirers. Other scholars consider different criteria to describe rigor in qualitative research process. 7 The 2 concepts of reliability and validity have been operationalized eloquently in quantitative texts but at the same time were deemed not pertinent to qualitative inquiries in the 1990s. Meticulous attention to the reliability and validity of research studies is particularly vital in qualitative work, where the researcher's subjectivity can so readily cloud the interpretation of the data and where research findings are often questioned or viewed with skepticism by the scientific community (Brink, 1993).

This article will discuss the issue of rigor in relation to qualitative research and further illustrate the process using a phenomenological study as an exemplar based on Lincoln and Guba's 1 (1985) techniques. This approach will clarify and define some of these complex concepts. There are numerous articles about trustworthiness in the literature that are too complex, confusing, and full of jargon. Some of these published articles also discuss rigor vis-à-vis reliability and validity in a very complicated way. Rigor will be first defined followed by how “reliability and validity” should be applied to qualitative research methods during the inquiry (constructive) rather than only post hoc evaluation. Strategies to attain reliability and validity will be described including the criteria and techniques for ensuring its attainment in a study. This discussion will critically focus on the misuse or nonuse of the concept of reliability and validity in qualitative inquiries, reestablish its importance, and relate both to the concept of rigor. Reflecting on my own research experience, recommendations for the renewed use of the concept of reliability and validity in qualitative research will be presented.

RIGOR VERSUS TRUSTWORTHINESS

Rigor of qualitative research continues to be challenged even now in the 21st century—from the very idea that qualitative research alone is open to questions, so with the terms rigor and trustworthiness . It is critical to understand rigor in research. Rigor is simply defined as the quality or state of being very exact, careful, or with strict precision 8 or the quality of being thorough and accurate. 9 The term qualitative rigor itself is an oxymoron, considering that qualitative research is a journey of explanation and discovery that does not lend to stiff boundaries. 10

Rigor and truth are always of concern for qualitative research. 11 Rigor has also been used to express attributes related to the qualitative research process. 12,13 Per Morse et al 4 (2002), without rigor, research is worthless, becomes fiction, and loses its use. The authors further defined rigor as the strength of the research design and the appropriateness of the method to answer the questions. It is expected that qualitative studies be conducted with extreme rigor because of the potential of subjectivity that is inherent in this type of research. This is a more difficult task when dealing with narratives and people than numbers and statistics. 14 Davies and Dodd 13 (2002) refer rigor to the reliability and validity of research and that, inherent to the conception, the concept is a quantitative bias. Several researchers argued that reliability and validity pertain to quantitative research, which is unrelated or not pertinent to qualitative inquiry because it is aligned with the positivist view. 15 It is also suggested that a new way of looking at reliability and validity will ensure rigor in qualitative inquiry. 1,16 From Lincoln and Guba's crucial work in the 1980s, reliability and validity were replaced with the concept “trustworthiness.” Lincoln and Guba 1 (1985) were the first to address rigor in their model of trustworthiness of qualitative research. Trustworthiness is used as the central concept in their framework to appraise the rigor of a qualitative study.

Trustworthiness is described in different ways by researchers. Trustworthiness refers to quality, authenticity, and truthfulness of findings of qualitative research. It relates to the degree of trust, or confidence, readers have in results. 14 Yin 17 (1994) describes trustworthiness as a criterion to judge the quality of a research design. Trustworthiness addressed methods that can ensure one has carried out the research process correctly. 18 Manning 19 (1997) considered trustworthiness as parallel to the empiricist concepts of internal and external validity, reliability, and objectivity. Seale 20 (1999) asserted that trustworthiness of a research study is based on the concepts of reliability and validity. Guba 2 (1981), Guba and Lincoln 3 (1982), and Lincoln and Guba 1 (1985) refer to trustworthiness as something that evolved from 4 major concerns that relate to it in which the set of criteria were based on. Trustworthiness is a goal of the study and, at the same time, something to be judged during the study and after the research is conducted. The 4 major traditional criteria are summarized into 4 questions about truth value, applicability, consistency, and neutrality. From these, they proposed 4 analogous terms within the naturalistic paradigm to replace the rationalistic terms: credibility, transferability, dependability, and confirmability. 1 For each of these 4 naturalistic terms are research activities or steps that the inquirer should be engage in to be able to safeguard or satisfy each of the previously mentioned criteria and thus attain trustworthiness (Supplemental Digital Content 1, https://links.lww.com/DCCN/A18 ). Guba and Lincoln 1 (1985) stated:

The criteria aid inquirers in monitoring themselves and in guiding activities in the field, as a way of determining whether or not various stages in the research are meeting standards for quality and rigor. Finally, the same criteria may be used to render ex-post facto judgments on the products of research, including reports, case studies, or proposed publications.

Standards and checklist were developed in the 1990s based on Lincoln and Guba's 1 (1985) established criteria, which were then discarded in favor of principles. 21 These standards and checklists consisted of long list of strategies used by qualitative researchers, which were thought to cause harm because of the confusion on which strategies were appropriate for certain designs or what type of naturalistic inquiry is being evaluated. Thus, researchers interpreted missing data as faults and flaws. 21 Morse 21 (2012) further claimed that these standards became the qualitative researchers' “worst enemies” and such an approach was not appropriate. Guba and Lincoln 18 (1989) later proposed a set of guidelines for post hoc evaluation of a naturalistic inquiry to ensure trustworthiness based on the framework of naturalism and constructivism and beyond the conventional methodological ideas. The aspects of their criteria have been fundamental to development of standards used to evaluate the quality of qualitative inquiry. 4

THE RIGOR DEBATES: TRUSTWORTHINESS OR RELIABILITY AND VALIDITY?

A research endeavor, whether quantitative or qualitative, is always evaluated for its worth and merits by peers, experts, reviewers, and readers. Does this mean that a study is differentiated between “good” and “bad”? What determines a “good” from a “bad” inquiry? For a quantitative study, this would mean determining the reliability and validity, and for qualitative inquiries, this would mean determining rigor and trustworthiness. According to Golafshani 22 (2003), if the issues of reliability, validity, trustworthiness, and rigor are meant to differentiating a “good” from “bad” research, then testing and increasing the reliability, validity, trustworthiness, and rigor will be important to the research in any paradigm. However, do reliability and validity in quantitative research equate totally to rigor and trustworthiness in qualitative research? There are many ways to assess the “goodness” of a naturalistic inquiry. Guba and Lincoln 18 (1989) asked, “‘What standards ought apply?’… goodness criteria like paradigms are rooted in certain assumptions. Thus, it is not appropriate to judge constructivist evaluations by positivistic criteria or standards or vice versa. To each its proper and appropriate set.”

Reliability and validity are analogues and are determined differently than in quantitative inquiry. 21 The nature and purpose of the quantitative and qualitative traditions are also different that it is erroneous to apply the same criteria of worthiness or merit. 23,24 The qualitative researcher should not focus on quantitatively defined indicators of reliability and validity, but that does not mean that rigorous standards are not appropriate for evaluating findings. 11 Evaluation, like democracy, is a process that, to be at its best, depends on the application of enlightened and informed self-interest. 18 Agar 24 (1986), on the other hand, suggested that terms such as reliability and validity are comparative with the quantitative view and do not fit the details of qualitative research. A different language is needed to fit the qualitative view. From Leininger 25 (1985), Krefting 23 (1991) asserted that addressing reliability and validity in qualitative research is such a different process that quantitative labels should not be used. The incorrect application of the qualitative criteria of rigor to studies is as problematic as the application of inappropriate quantitative criteria. 23 Smith 26 (1989) argued that, for qualitative research, this means that the basis of truth or trustworthiness becomes a social agreement. He emphasizes that what is judged true or trustworthy is what we can agree, conditioned by time and place, and is true or trustworthy. Validity standards in qualitative research are also even more challenging because of the necessity to incorporate rigor and subjectivity, as well as creativity into the scientific process. 27 Furthermore, Leininger 25 (1985) claimed that it is not whether the data are reliable or valid but how the terms reliability and validity are defined. Aside from the debate whether reliability and validity criteria should be used similarly in qualitative inquiries, there is also an issue of not using the concepts at all in naturalistic studies.

Designing a naturalistic inquiry is very different from a traditional quantitative notion of design and that defining a “good” qualitative inquiry is controversial and has gone through many changes. 21 First is the confusion on the use of terminologies “rigor” and “trustworthiness.” Morse 28 (2015) suggested that it is time to return to the terminology of mainstream social science and to use “rigor” rather than “trustworthiness.” Debates also continue about why some qualitative researchers do not use the concept of reliability and validity in their studies referring to Lincoln and Guba's 1 (1985) criteria for trustworthiness, namely, transferability, dependability, confirmability, and credibility. Morse 28 (2015) further suggested replacing these criteria to reliability, validity, and generalizability. The importance and centrality of reliability and validity to qualitative inquiries have in some way been disregarded even in the current times. Researchers from the United Kingdom and Europe continue to do so but not much so in North America. 4 According to Morse 21 (2012), this gives the impression that these concepts are of no concern to qualitative research. Morse 29 (1999) stated, “Is the terminology worth making a fuzz about?”, when Lincoln and Guba 1 (1985) described trustworthiness and reliability and validity as analogs. Morse 29 (1999) further articulated that:

To state that reliability and validity are not pertinent to qualitative inquiry places qualitative research in the realm of being not reliable and not valid. Science is concerned with rigor, and by definition, good rigorous research must be reliable and valid. If qualitative research is unreliable and invalid, then it must not be science. If it is not science, then why should it be funded, published, implemented, or taken seriously?

RELIABILITY AND VALIDITY IN QUALITATIVE RESEARCH

Reliability and validity should be taken into consideration by qualitative inquirers while designing a study, analyzing results, and judging the quality of the study, 30 but for too long, the criteria used for evaluating rigor are applied after a research is completed—a considerably wrong tactic. 4 Morse and colleagues 4 (2002) argued that, for reliability and validity to be actively attained, strategies for ensuring rigor must be built into the qualitative research process per se not to be proclaimed only at the end of the inquiry. The authors suggest that focusing on strategies to establish rigor at the completion of the study (post hoc), rather than during the inquiry, exposes the investigators to the risk of missing and addressing serious threats to the reliability and validity until it is too late to correct them. They further asserted that the interface between reliability and validity is important especially for the direction of the analysis process and the development of the study itself.

Reliability

In the social sciences, the whole notion of reliability in and of itself is problematic. 31 The scientific aspect of reliability assumes that repeated measures of a phenomenon (with the same results) using objective methods establish the truth of the findings. 32-35 Merriam 36 (1995) stated that, “The more times the findings of a study can be replicated, the more stable or reliable the phenomenon is thought to be.” In other words, it is the idea of replicability, 22,34,37 repeatability, 21,22,26,30,31,36,38-40 and stability of results or observation. 25,39,41 The issues are that human behaviors and interactions are never static or the same. Measurements and observations can also be repeatedly wrong. Furthermore, researchers have argued that the concept reliability is misleading and has no relevance in qualitative research related to the notion of “measurement method,” as in quantitative studies. 40,42 It is a fact that quantitative research is supported by the positivist or scientific paradigm that regards the world as made up of observable, measurable facts. Qualitative research, on the other hand, produces findings not arrived at by means of statistical procedures or other means of quantification. On the basis of the constructivist paradigm, it is a naturalistic inquiry that seeks to understand phenomena in context-specific settings in which the researcher does not attempt to manipulate the phenomenon of interest. 23 If reliability is used as a criterion in qualitative research, it would mean that the study is “not good.” A thorough description of the entire research process that allows for intersubjectivity is what indicates good quality when using qualitative methodology. Reliability is based on consistency and care in the application of research practices, which are reflected in the visibility of research practices, analysis, and conclusions, reflected in an open account that remains mindful of the partiality and limits of the research findings. 13 Reliability and similar terms are presented in Supplemental Digital Content 2 (see Supplemental Digital Content 2, https://links.lww.com/DCCN/A19 ).

Validity is broadly defined as the state of being well grounded or justifiable, relevant, meaningful, logical, confirming to accepted principles or the quality of being sound, just, and well founded. 8 The issues surrounding the use and nature of the term validity in qualitative research are controversial and many. It is a highly debated topic both in social and educational research and is still often a subject of debate. 43 The traditional criteria for validity find their roots in a positivist tradition, and to an extent, positivism has been defined by a systematic theory of validity. 22 Validity is rooted from empirical conceptions as universal laws, evidence, objectivity, truth, actuality, deduction, reason, fact, and mathematical data, to name only a few. Validity in research is concerned with the accuracy and truthfulness of scientific findings. 44 A valid study should demonstrate what actually exists and is accurate, and a valid instrument or measure should actually measure what it is supposed to measure. 5,22,29,31,42,45

Novice researchers can become easily perplexed in attempting to understand the notion of validity in qualitative inquiry. 44 There is a multiple array of terms similar to validity in the literature, which the authors equate to same such as authenticity, goodness, adequacy, trustworthiness, verisimilitude, credibility, and plausibility. 1,45-51 Validity is not a single, fixed, or universal concept but rather a contingent construct, inescapably grounded in the processes and intentions of particular research methodologies. 39 Some qualitative researchers have argued that the term validity is not applicable to qualitative research and have related it to terms such as quality, rigor, and trustworthiness. 1,13,22,38,42,52-54 I argue that the concepts of reliability and validity are overarching constructs that can be appropriately used in both quantitative and qualitative methodologies. To validate means to investigate, to question, and to theorize, which are all activities to ensure rigor in a qualitative inquiry. For Leininger 25 (1985), the term validity in a qualitative sense means gaining knowledge and understanding of the nature (ie, the meaning, attributes, and characteristics) of the phenomenon under study. A qualitative method seeks for a certain quality that is typical for a phenomenon or that makes the phenomenon different than others.

Some naturalistic inquirers agree that assuring validity is a process whereby ideals are sought through attention to specified criteria, and appropriate techniques are used to address any threats to validity of a naturalistic inquiry. However, other researchers argue that procedures and techniques are not an assurance of validity and will not necessarily produce sound data or credible conclusions. 38,48,55 Thus, some argued that they should abandon the concept of validity and seek alternative criteria with which to judge their work. Criteria are the standards or rules to be upheld as ideals in qualitative research on which a judgment or decisions may be based, 4,56 whereas the techniques are the methods used to diminish identified validity threats. 56 Criteria, for some researchers, are used to test the quality of the research design, whereas for some, they are the goal of the study. There is also the trend to treat standards, goals, and criteria synonymously. I concur with Morse 29 (1999) that introducing parallel terminology and criteria diminishes qualitative inquiry from mainstream science and scientific legitimacy. The development of alternative criteria compromises the issue of rigor. We must work to have a consensus toward criteria and terminology that are used in mainstream science and how it is attained within the qualitative inquiry during the research process rather than at the end of the study. Despite all these, researchers developed validity criteria and techniques during the years. A synthesis of validity criteria development is summarized in Supplemental Digital Content 3 (see Supplemental Digital Content 3, https://links.lww.com/DCCN/A20 ). The techniques for demonstrating validity are presented in Supplemental Digital Content 4 (see Supplemental Digital Content 4, https://links.lww.com/DCCN/A21 ).

Reliability and Validity as Means in Ensuring the Quality of Findings of a Phenomenological Study in Intensive Care Unit

Reliability and validity are 2 factors that any qualitative researcher should be concerned about while designing a study, analyzing results, and judging its quality. Just as the quantitative investigator must attend to the question of how external and internal validity, reliability, and objectivity will be provided for in the design, so must the naturalistic inquirer arrange for credibility, transferability, dependability, and confirmability. 1 Lincoln and Guba 1 (1985) clearly established these 4 criteria as benchmarks for quality based on the identification of 4 aspects of trustworthiness that are relevant to both quantitative and qualitative studies, which are truth value, applicability, consistency, and neutrality. Guba 2 (1981) stated, “It is to these concerns that the criteria must speak.”

Rigor of a naturalistic inquiry such as phenomenology may be operationalized using the criteria of credibility, transferability, dependability, and confirmability. This phenomenological study aimed to understand and illuminate the meaning of the phenomenon of the lived experiences of patients, their family members, and the nurses during critical illness in the intensive care unit (ICU). From Lincoln and Guba 1 (1985), I first asked the question, “How can I persuade my audience that the research findings of my inquiry are worth paying attention to, and worth taking account of?” My answers to these questions were based on the identified 4 criteria set forth by Lincoln and Guba 1 (1985).

Credibility, the accurate and truthful depiction of a participant's lived experience, was achieved in this study through prolonged engagement and persistent observation to learn the context of the phenomenon in which it is embedded and to minimize distortions that might creep into the data. To achieve this, I spent 6 months with nurses, patients, and their families in the ICU to become oriented to the situation and also to build trust and rapport with the participants. Peer debriefing was conducted through meetings and discussions with an expert qualitative researcher to allow for questions and critique of field journals and research activities. Triangulation was achieved by cross-checking the data and interpretations within and across each category of participants by 2 qualitative researchers. Member checks were accomplished by constantly checking data and interpretations with the participants from which data were solicited.

Transferability was enhanced by using purposive sampling method and providing a thick description and a robust data with a wide possible range of information through the detailed and accurate descriptions of the patients, their family members, and the nurses' lived ICU experiences and by continuously returning to the texts. In this study, recruitment of participants and data collection continued until the data are saturated and complete and replicate. According to Morse et al 4 (2002), interviewing additional participants is for the purpose of increasing the scope, adequacy, and appropriateness of the data. I immersed myself into the phenomenon to know, describe, and understand it fully, comprehensively, and thoroughly. Special care was given to the collection, identification, and analysis of all data pertinent to the study. The audiotaped data were meticulously transcribed by a professional transcriber for future scrutiny. During the analysis phase, every attempt was made to document all aspects of the analysis. Analysis in qualitative research refers to the categorization and ordering of information in such a way as to make sense of the data and to writing a final report that is true and accurate. 36 Every effort was made to coordinate methodological and analytical materials. After I categorized and was able to make sense of the transcribed data, all efforts were exhausted to illuminate themes and descriptors as they emerge.

Lincoln and Guba 1 (1985) use “dependability” in qualitative research, which closely corresponds to the notion of “reliability” in quantitative research. Dependability was achieved by having 2 expert qualitative nursing researchers review the transcribed material to validate the themes and descriptors identified. To be able to validate my findings related to the themes, a doctoral-prepared nursing colleague was asked to review some of the transcribed materials. Any new themes and descriptors illuminated by my colleague were acknowledged and considered. It was then compared with my own thematic analysis from the entire participant's transcribed data. If the theme identified by the colleague did not appear in my own thematic analysis, it was agreed by both analysts not to use the said theme. It was my goal that both analysts agree on the findings related to themes and meanings within the transcribed material.

Confirmability was met by maintaining a reflexive journal during the research process to keep notes and document introspections daily that would be beneficial and pertinent during the study. An audit trail also took place to examine the processes whereby data were collected and analyzed and interpretations were made. The audit trail took the form of documentation (the actual interview notes taken) and a running account of the process (my daily field journal). I maintained self-awareness of my role as the sole instrument of this study. After each interview, I retired in 1 private room to document additional perceptions and recollections from the interviews (Supplemental Digital Content 5, https://links.lww.com/DCCN/A22 ).

Through reflexivity and bracketing, I was always on guard of my own biases, assumptions, beliefs, and presuppositions that I might bring to the study but was also aware that complete reduction is not possible. Van Manen 44 (1990) stated that “if we simply try to forget or ignore what we already know, we may find that the presuppositions persistently creep back into our reflections.” During data collection and analysis, I made my orientation and preunderstanding of critical illness and critical care explicit but held them deliberately at bay and bracketed them. Aside from Lincoln and Guba's 1 (1985) 4 criteria for trustworthiness, a question arises as to the reliability of the researcher as the sole instrument of the study.

Reliability related to the researcher as the sole instrument who conducted the data collection and analysis is a limitation of any phenomenological study. The use of humans as instruments is not a new concept. Lincoln and Guba 1 (1985) articulated that humans uniquely qualify as the instrument of choice for naturalistic inquiry. Some of the giants of conventional inquiry have recognized that humans can provide data very nearly as reliable as that produced by “more” objective means. These are formidable characteristics, but they are meaningless if the human instrument is not also trustworthy. However, no human instrument is expected to be perfect. Humans have flaws, and errors could be committed. When Lincoln and Guba 1 (1985) asserted that qualitative methods come more easily to hand when the instrument is a human being, they mean that the human as instrument is inclined toward methods that are extensions of normal activities. They believe that the human will tend therefore toward interviewing, observing, mining available documents and records, taking account of nonverbal cues, and interpreting inadvertent unobtrusive measures. All of which are complex tasks. In addition, one would not expect an individual to function adequately as human instruments without an extensive background or training and experience. This study has reliability in that I have acquired knowledge and the required training for research at a doctoral level with the professional and expert guidance of a mentor. As Lincoln and Guba 1 (1985) said, “Performance can be improved…when that learning is guided by an experienced mentor, remarkable improvements in human instrumental performance can be achieved.” Whereas reliability in quantitative research depends on instrument construction, in qualitative research, the researcher is the instrument of the study. 31 A reliable research is a credible research. Credibility of a qualitative research depends on the ability and effort of the researcher. 22 We have established that a study can be reliable without being valid, but a study cannot be valid without being reliable.

Establishing validity is a major challenge when a qualitative research project is based on a single, cross-sectional, unstructured interview as the basis for data collection. How do I make judgments about the validity of the data? In qualitative research, the validity of the findings is related to the careful recording and continual verification of the data that the researcher undertakes during the investigative practice. If the validity or trustworthiness can be maximized or tested, then more credible and defensible result may lead to generalizability as the structure for both doing and documenting high-quality qualitative research. Therefore, the quality of a research is related to generalizability of the result and thereby to the testing and increasing of the validity or trustworthiness of the research.

One potential threat to validity that researchers need to consider is researcher bias. Researcher bias is frequently an issue because qualitative research is open and less structured than quantitative research. This is because qualitative research tends to be exploratory. Researcher bias tends to result from selective observation and selective recording of information and from allowing one's personal views and perspectives to affect how data are interpreted and how the research is conducted. Therefore, it is very important that the researchers are aware of their own perceptions and opinions because they may taint their research findings and conclusions. I brought all past experiences and knowledge into the study but learned to set aside my own strongly held perceptions, preconceptions, and opinions. I truly listened to the participants to learn their stories, experiences, and meanings.

The key strategy used to understand researcher bias is called reflexivity. Reflexivity means that the researchers actively engage in critical self-reflection about their potential biases and predispositions that they bring to the qualitative study. Through reflexivity, researchers become more self-aware and monitor and attempt to control their biases. Phenomenological researchers can recognize that their interpretation is correct because the reflective process awakens an inner moral impulse. 4,59 I did my best to be always on guard of my own biases, preconceptions, and assumptions that I might bring to this study. Bracketing was also applied.

Husserl 60 (1931) has made some key conceptual elaborations, which led him to assert that an attempt to hold a previous belief about the phenomena under study in suspension to perceive it more clearly is needed in phenomenological research. This technique is called bracketing. Bracketing is another strategy used to control bias. Husserl 60 (1931) explained further that phenomenological reduction is the process of defining the pure essence of a psychological phenomenon. Phenomenological reduction is a process whereby empirical subjectivity is suspended, so that pure consciousness may be defined in its essential and absolute “being.” This is accomplished by a method of bracketing empirical data away from consideration. Bracketing empirical data away from further investigation leaves pure consciousness, pure phenomena, and pure ego as the residue of phenomenological reduction. Husserl 60 (1931) uses the term epoche (Greek word for “a cessation”) to refer to this suspension of judgment regarding the true nature of reality. Bracketed judgment is an epoche or suspension of inquiry, which places in brackets whatever facts belong to essential “being.”

Bracketing was conducted to separate the assumptions and biases from the essences and therefore achieve an understanding of the phenomenon as experienced by the participants of the study. The collected and analyzed data were presented to the participants, and they were asked whether the narrative is accurate and a true reflection of their experience. My interpretation and descriptions of the narratives were presented to the participants to achieve credibility. They were given the opportunity to review the transcripts and modify it if they wished to do so. As I was the one who served as the sole instrument in obtaining data for this phenomenological study, my goal was that my perceptions would reflect the participant's ICU experiences and that the participants would be able to see their lived experience through the researcher's eyes. Because qualitative research designs are flexible and emergent in nature, there will always be study limitations.

Awareness of the limitations of a research study is crucial for researchers. The purpose of this study was to understand the ICU experiences of patients, their family members, and the nurses during critical illness. One limitation of this phenomenological study as a naturalistic inquiry was the inability of the researcher to fully design and provide specific ideas needed for the study. According to Lincoln and Guba 1 (1985), naturalistic studies are virtually impossible to design in any definitive way before the study is actually undertaken. The authors stated:

Designing a naturalistic study means something very different from the traditional notion of “design”—which as often as not meant the specification of a statistical design with its attendant field conditions and controls. Most of the requirements normally laid down for a design statement cannot be met by naturalists because the naturalistic inquiry is largely emergent.

Within the naturalistic paradigm, designs must be emergent rather than preordinate because (1) meaning is determined by context to such a great extent. For this particular study, the phenomenon and context were the experience of critical illness in the ICU; (2) the existence of multiple realities constrains the development of a design based on only 1 (the investigator's) construction; (3) what will be learned at a site is always dependent on the interaction between the investigator and the context, and the interaction is also not fully predictable; and (4) the nature of mutual shapings cannot be known until they are witnessed. These factors underscore the indeterminacy under which naturalistic inquirer functions. The design must therefore be “played by ear”; it must unfold, cascade, and emerge. It does not follow, however, that, because not all of the elements of the design can be prespecified in a naturalistic inquiry, none of them can. Design in the naturalistic sense means planning for certain broad contingencies without however indicating exactly what will be conducted on relation to each. 1

Reliability and validity are such fundamental concepts that should be continually operationalized to meet the conditions of a qualitative inquiry. Morse et al 4,29 (2002) articulated that “by refusing to acknowledge the centrality of reliability and validity in qualitative methods, qualitative methodologists have inadvertently fostered the default notion that qualitative research must therefore be unreliable and invalid, lacking in rigor, and unscientific.” Sparkes 59 (2001) asserted that Morse et al 4,26 (2002) is right in warning us that turning our backs on such fundamental concepts as validity could cost us dearly. This will in turn affect how we mentor novices, early career researchers, and doctoral students in their qualitative research works.

Reliability is inherently integrated and internally needed to attain validity. 1,26 I concur with the use of the term rigor rather than trustworthiness in naturalistic studies. I have also discussed that I accede that strategies for ensuring rigor must be built into the qualitative research process per se rather than evaluated only after the inquiry is conducted. Threats to reliability and validity cannot be actively addressed by using standards and criteria applied at the end of the study. Ensuring rigor must be upheld by the researcher during the investigation rather than the external judges of the completed study. Whether a study is quantitative or qualitative, rigor is a desired goal that is met through the inclusion of different philosophical perspectives inherent in a qualitative inquiry and the strategies that are specific to each methodological approach including the verification techniques to be observed during the research process. It also involves the researcher's creativity, sensitivity, flexibility, and skill in using the verification strategies that determine the reliability and validity of the evolving study.

Some naturalistic inquirers agree that assuring validity is a process whereby ideals are sought through attention to specified criteria, and appropriate techniques are used to address any threats to validity of a naturalistic inquiry. However, other researchers argue that procedures and techniques are not an assurance of validity and will not necessarily produce sound data or credible conclusions. 38,48,55 Thus, some argued that they should abandon the concept of validity and seek alternative criteria with which to judge their work.

Lincoln and Guba's 1 (1985) standards of validity demonstrate the necessity and convenience of overarching principles to all qualitative research, yet there is a need for a reconceptualization of criteria of validity in qualitative research. The development of validity criteria in qualitative research poses theoretical issues, not simply technical problems. 60 Whittemore et al 58 (2001) explored the historical development of validity criteria in qualitative research and synthesized the findings that reflect a contemporary reconceptualization of the debate and dialogue that have ensued in the literature during the years. The authors further presented primary (credibility, authenticity, criticality, and integrity) and secondary (explicitness, vividness, creativity, thoroughness, congruence, and sensitivity) validity criteria to be used in the evaluative process. 56 Before the work of Whittemore and colleagues, 58 Creswell and Miller 48 (2000) asserted that the constructivist lens and paradigm choice should guide validity evaluation and procedures from the perspective of the researcher (disconfirming evidence), the study participants (prolonged engagement in the field), and external reviewers/readers (thick, rich description). Morse et al 4 in 2002 presented 6 major evaluation criteria for validity and asserted that they are congruent and are appropriate within the philosophy of the qualitative tradition. These 6 criteria are credibility, confirmability, meaning in context, recurrent patterning, saturation, and transferability. Synthesis of validity criteria is presented in Supplemental Digital Content 3 (see Supplemental Digital Content 3, https://links.lww.com/DCCN/A20 ).

Common validity techniques in qualitative research refer to design consideration, data generation, analytic procedures, and presentation. 56 First is the design consideration. Developing a self-conscious design, the paradigm assumption, the purposeful choice of small sample of informants relevant to the study, and the use of inductive approach are some techniques to be considered. Purposive sampling enhances the transferability of the results. Interpretivist and constructivist inquiry follows an inductive approach that is flexible and emergent in design with some uncertainty and fluidity within the context of the phenomenon of interest 56,58 and not based on a set of determinate rules. 61 The researcher does not work with a priori theory; rather, these are expected to emerge from the inquiry. Data are analyzed inductively from specific, raw units of information to subsuming categories to define questions that can be followed up. 1 Qualitative studies also follow a naturalistic and constructivist paradigm. Creswell and Miller 48 (2000) suggest that the validity is affected by the researchers' perception of validity in the study and their choice of paradigm assumption. Determining fit of paradigm to focus is an essential aspect of a naturalistic inquiry. 1 Paradigms rest on sets of beliefs called axioms. 1 On the basis of the naturalistic axioms, the researcher should ask questions related to multiplicity or complex constructions of the phenomenon, the degree of investigator-phenomenon interaction and the indeterminacy it will introduce into the study, the degree of context dependence, whether values are likely to be crucial to the outcome, and the constraints that may be placed on the researcher by a variety of significant others. 1

Validity during data generation is evaluated through the researcher's ability to articulate data collection decisions, demonstrate prolonged engagement and persistent observation, provide verbatim transcription, and achieve data saturation. 56 Methods are means to collect evidence to support validity, and this refers to the data obtained by considering a context for a purpose. The human instrument operating in an indeterminate situation falls back on techniques such as interview, observation, unobtrusive measures, document and record analysis, and nonverbal cues. 1 Others remarked that rejecting methods or technical procedures as assurance of truth, thus validity of a qualitative study, lies in the skills and sensitivities of the researchers and how they use themselves as a knower and an inquirer. 57,62 The understanding of the phenomenon is valid if the participants are given the opportunity to speak freely according to their own knowledge structures and perceptions. Validity is therefore achieved when using the method of open-ended, unstructured interviews with strategically chosen participants. 42 We also know that a thorough description of the entire research process enabling unconditional intersubjectivity is what indicates good quality when using a qualitative method. This enhances a clearer and better analysis of data.

Analytical procedures are vital in qualitative research. 56 Not very much can be said about data analysis in advance of a qualitative study. 1 Data analysis is not an inclusive phase that can be marked out as occurring at some singular time during the inquiry. 1 It begins from the very first data collection to facilitate the emergent design and grounding of theory. Validity in a study thus is represented by truthfulness of findings after a careful analysis. 56 Consequently, qualitative researchers seek to illuminate and extrapolate findings to similar situations. 22,63 It is a fact that the interpretations of any given social phenomenon may reflect, in part, the biases and prejudices the interpreters bring to the task and the criteria and logic they follow in completing it. 64 In any case, individuals will draw different conclusions to the debate surrounding validity and will make different judgments as a result. 50 There is a wide array of analytic techniques that the qualitative researcher can choose from based on the contextual factors that will help contribute to the decision as to which technique will optimally reflect specific criteria of validity. 65 Presentation of findings is accomplished by providing an audit trail and evidence that support interpretations, acknowledging the researcher's perspective and providing thick descriptions. Morse et al 4 in 2002 set forth strategies for ensuring validity that include investigator responsiveness and verification through methodological coherence, theoretical sampling and sampling adequacy, an active analytic stance, and saturation. The authors further stated that “these strategies, when used appropriately, force the researcher to correct both the direction of the analysis and the development of the study as necessary, thus ensuring reliability and validity of the completed project (p17). Recently in 2015, Morse 28 presented that the strategies for ensuring validity in a qualitative study are prolonged engagement, persistent observation, thick and rich description, negative case analysis, peer review or debriefing, clarifying researcher's bias, member checking, external audits, and triangulation. These strategies can be upheld with the help of an expert mentor who can in turn guide and affect the reliability and validity of early career researchers and doctoral students' qualitative research works. Techniques for demonstrating validity are summarized in Supplemental Digital Content 4 (see Supplemental Digital Content 4, https://links.lww.com/DCCN/A21 ).

Qualitative researchers and students alike must be proactive and take responsibility in ensuring the rigor of a research study. A lot of times, rigor is at the backseat in some researchers and doctoral students' work related to their novice abilities, lack of proper mentorship, and issues with time and funding. Students should conduct projects that are smaller in scope guided by an expert naturalistic inquirer to come up with the product with depth and, at the same time, gain the grounding experience necessary to become an excellent researcher. Attending to rigor throughout the research process will have important ramifications for qualitative inquiry. 4,26

Qualitative research is not intended to be scary or beyond the grasp of novices and doctoral students. Conducting a naturalistic inquiry is an experience of exploration, discovery, description, and understanding of a phenomenon that transcends one's own research journey. Attending to the rigor of qualitative research is a vital part of the investigative process that offers critique and thus further development of the science.

• Cited Here |
• Google Scholar

Phenomenology; Qualitative research; Reliability; Rigor; Validity

Supplemental Digital Content

• DCCN_2017_04_11_CYPRESS_DCCN-D-16-00060_SDC1.pdf; [PDF] (3 KB)
• DCCN_2017_04_11_CYPRESS_DCCN-D-16-00060_SDC2.pdf; [PDF] (4 KB)
• DCCN_2017_04_11_CYPRESS_DCCN-D-16-00060_SDC3.pdf; [PDF] (78 KB)
• DCCN_2017_04_11_CYPRESS_DCCN-D-16-00060_SDC4.pdf; [PDF] (70 KB)
• DCCN_2017_04_11_CYPRESS_DCCN-D-16-00060_SDC5.pdf; [PDF] (4 KB)
• + Favorites
• View in Gallery

Readers Of this Article Also Read

Family presence during resuscitation: the education needs of critical care..., increasing access to palliative care services in the intensive care unit, educational interventions to improve support for family presence during..., nursing practices and policies related to family presence during resuscitation.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals

You are here

• Volume 18, Issue 2
• Issues of validity and reliability in qualitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Helen Noble 1 ,
• Joanna Smith 2
• 1 School of Nursing and Midwifery, Queens's University Belfast , Belfast , UK
• 2 School of Human and Health Sciences, University of Huddersfield , Huddersfield , UK
• Correspondence to Dr Helen Noble School of Nursing and Midwifery, Queens's University Belfast, Medical Biology Centre, 97 Lisburn Rd, Belfast BT9 7BL, UK; helen.noble{at}qub.ac.uk

https://doi.org/10.1136/eb-2015-102054

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Evaluating the quality of research is essential if findings are to be utilised in practice and incorporated into care delivery. In a previous article we explored ‘bias’ across research designs and outlined strategies to minimise bias. 1 The aim of this article is to further outline rigour, or the integrity in which a study is conducted, and ensure the credibility of findings in relation to qualitative research. Concepts such as reliability, validity and generalisability typically associated with quantitative research and alternative terminology will be compared in relation to their application to qualitative research. In addition, some of the strategies adopted by qualitative researchers to enhance the credibility of their research are outlined.

Are the terms reliability and validity relevant to ensuring credibility in qualitative research?

Although the tests and measures used to establish the validity and reliability of quantitative research cannot be applied to qualitative research, there are ongoing debates about whether terms such as validity, reliability and generalisability are appropriate to evaluate qualitative research. 2–4 In the broadest context these terms are applicable, with validity referring to the integrity and application of the methods undertaken and the precision in which the findings accurately reflect the data, while reliability describes consistency within the employed analytical procedures. 4 However, if qualitative methods are inherently different from quantitative methods in terms of philosophical positions and purpose, then alterative frameworks for establishing rigour are appropriate. 3 Lincoln and Guba 5 offer alternative criteria for demonstrating rigour within qualitative research namely truth value, consistency and neutrality and applicability. Table 1 outlines the differences in terminology and criteria used to evaluate qualitative research.

• View inline

Terminology and criteria used to evaluate the credibility of research findings

What strategies can qualitative researchers adopt to ensure the credibility of the study findings?

Unlike quantitative researchers, who apply statistical methods for establishing validity and reliability of research findings, qualitative researchers aim to design and incorporate methodological strategies to ensure the ‘trustworthiness’ of the findings. Such strategies include:

Accounting for personal biases which may have influenced findings; 6

Acknowledging biases in sampling and ongoing critical reflection of methods to ensure sufficient depth and relevance of data collection and analysis; 3

Meticulous record keeping, demonstrating a clear decision trail and ensuring interpretations of data are consistent and transparent; 3 , 4

Establishing a comparison case/seeking out similarities and differences across accounts to ensure different perspectives are represented; 6 , 7

Including rich and thick verbatim descriptions of participants’ accounts to support findings; 7

Demonstrating clarity in terms of thought processes during data analysis and subsequent interpretations 3 ;

Engaging with other researchers to reduce research bias; 3

Respondent validation: includes inviting participants to comment on the interview transcript and whether the final themes and concepts created adequately reflect the phenomena being investigated; 4

Data triangulation, 3 , 4 whereby different methods and perspectives help produce a more comprehensive set of findings. 8 , 9

Table 2 provides some specific examples of how some of these strategies were utilised to ensure rigour in a study that explored the impact of being a family carer to patients with stage 5 chronic kidney disease managed without dialysis. 10

Strategies for enhancing the credibility of qualitative research

In summary, it is imperative that all qualitative researchers incorporate strategies to enhance the credibility of a study during research design and implementation. Although there is no universally accepted terminology and criteria used to evaluate qualitative research, we have briefly outlined some of the strategies that can enhance the credibility of study findings.

• Sandelowski M
• Lincoln YS ,
• Barrett M ,
• Mayan M , et al
• Greenhalgh T
• Lingard L ,

Twitter Follow Joanna Smith at @josmith175 and Helen Noble at @helnoble

Competing interests None.

Read the full text or download the PDF:

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Validity in qualitative research

Affiliation.

• 1 Yale University School of Nursing, USA.
• PMID: 11521609
• DOI: 10.1177/104973201129119299

Much contemporary dialogue has centered on the difficulty of establishing validity criteria in qualitative research. Developing validity standards in qualitative research is challenging because of the necessity to incorporate rigor and subjectivity as well as creativity into the scientific process. This article explores the extant issues related to the science and art of qualitative research and proposes a synthesis of contemporary viewpoints. A distinction between primary and secondary validity criteria in qualitative research is made with credibility, authenticity, criticality, and integrity identified as primary validity criteria and explicitness, vividness, creativity, thoroughness, congruence, and sensitivity identified as secondary validity criteria.

PubMed Disclaimer

Similar articles

• Myth 94: qualitative health researchers will agree about validity. Sparkes AC. Sparkes AC. Qual Health Res. 2001 Jul;11(4):538-52. doi: 10.1177/104973230101100409. Qual Health Res. 2001. PMID: 11521610
• [Reconsidering evaluation criteria regarding health care research: toward an integrative framework of quantitative and qualitative criteria]. Miyata H, Kai I. Miyata H, et al. Nihon Koshu Eisei Zasshi. 2006 May;53(5):319-28. Nihon Koshu Eisei Zasshi. 2006. PMID: 16813064 Japanese.
• Enhancing the quality and credibility of qualitative analysis. Patton MQ. Patton MQ. Health Serv Res. 1999 Dec;34(5 Pt 2):1189-208. Health Serv Res. 1999. PMID: 10591279 Free PMC article. Review.
• How will we know "good" qualitative research when we see it? Beginning the dialogue in health services research. Devers KJ. Devers KJ. Health Serv Res. 1999 Dec;34(5 Pt 2):1153-88. Health Serv Res. 1999. PMID: 10591278 Free PMC article. Review.
• Validity in qualitative health education research. FitzPatrick B. FitzPatrick B. Curr Pharm Teach Learn. 2019 Feb;11(2):211-217. doi: 10.1016/j.cptl.2018.11.014. Epub 2018 Nov 22. Curr Pharm Teach Learn. 2019. PMID: 30733020 Review.
• Are we being equitable enough? Lessons learned from sites lost in an implementation trial. Austin EJ, Chen J, Ferro L, Saxon AJ, Fortney JC, Curran GM, Ratzliff AD, Williams EC. Austin EJ, et al. Implement Res Pract. 2024 Aug 14;5:26334895241267023. doi: 10.1177/26334895241267023. eCollection 2024 Jan-Dec. Implement Res Pract. 2024. PMID: 39148956 Free PMC article.
• Nurturing ethical insight: exploring nursing students' journey to ethical competence. Hansen S, Hessevaagbakke E, Lindeflaten K, Elvan K, Lillekroken D. Hansen S, et al. BMC Nurs. 2024 Aug 15;23(1):568. doi: 10.1186/s12912-024-02243-x. BMC Nurs. 2024. PMID: 39148119 Free PMC article.
• Patients' attitudes toward negative appendectomies and surgery for suspected appendicitis: a qualitative interview study. Henriksen SR, Konradsen H, Rosenberg J, Fonnes S. Henriksen SR, et al. Surg Endosc. 2024 Sep;38(9):5130-5136. doi: 10.1007/s00464-024-11020-5. Epub 2024 Jul 22. Surg Endosc. 2024. PMID: 39039290
• Participants' views of ultra-low dose combination therapy for high blood pressure: a mixed-methods study from the QUARTET trial. Kovoor JG, Chow CK, Salam A, Webster R, Shiel L, Nelson MR, Affandi JS, Hay P, Burke M, Figtree GA, Usherwood T, Reid CM, Schlaich MP, Rodgers A, Atkins ER. Kovoor JG, et al. J Hum Hypertens. 2024 Jun;38(6):516-522. doi: 10.1038/s41371-024-00915-4. Epub 2024 May 14. J Hum Hypertens. 2024. PMID: 38744907 Free PMC article.
• Knowledge and use of art therapy for mental health treatment among clinical psychologists. Doku Asare EP, Kushitor SB, Sutherland EK, Ofori Boateng M, Manortey S. Doku Asare EP, et al. PLoS One. 2024 May 9;19(5):e0303246. doi: 10.1371/journal.pone.0303246. eCollection 2024. PLoS One. 2024. PMID: 38722981 Free PMC article.
• Search in MeSH

LinkOut - more resources

Full text sources.

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.58; Jan-Dec 2021

Processual Validity in Qualitative Research in Healthcare

Paulo hayashi, jr.

1 University of Campinas- UNICAMP, Limeira, Brazil

Gustavo Abib

2 Paraná Federal University, Curitiba, Brazil

Norberto Hoppen

3 Vale do Rio dos Sinos University, São Leopoldo, Rio Grande do Sul, Brazil

Lillian Daisy Gonçalves Wolff

Knowledge development has been continuously challenging. Qualitative research seems to be promising; however, there are difficulties and complexities involved, one of which is validity. Qualitative research is based on different paradigms, ontologies, theories, and methods, and validity assessment may vary. We argue that processual validity can positively influence qualitative health care research. Processual validity is a methodological construction that involves all research steps, including those before and after data collection and analysis. We selected a processual validity model and two cases to illustrate its use and demonstrate processual validity’s importance and applicability. One case explores the gap between medical education and patients’ needs in primary health care. Other studies focus on health care improvements in hospitals. Our results highlight the benefits of processual validity to ensure the transparency and reliability of the research process and provide evidence of the findings to positively influence thinking and the execution of qualitative research in health care.

What do we already know about this topic?

Although qualitative research represents a broad range of traditions and the research design and methods can differ significantly from one study to another, there is a growing consensus about preoccupation with and adoption of rigorous quality criteria. Some scholars prefer to treat qualitative validity in the same way as quantitative validity. Other scholars prefer radical positions, such as eschewing validity. However, most qualitative researchers prefer a balanced method, where it is possible to guarantee some level of quality and flexibility. However, processual validity has only recently been treated as a scientific base. Thus, every stage of qualitative research is significant, and a processual approach to ensure validity can influence the quality of the evidence generated.

How does your research contribute to the field?

Understanding and ensuring research quality are crucial for science and practice. Our research provides additional insights for enhancing the credibility of qualitative research. Health care researchers that adopt the processual validity approach can build ex-ante field research validity, validity at moments of truth (during fieldwork), and ex-post field research validity.

What are your research implications for theory, practice, or policy?

The proposed and assessed processual validity approach is intended to deal with different perspectives of qualitative studies, with diverse quality dimensions resulting from various epistemological and ontological positions. In maintaining the flexibility, these different views on doing qualitative research require, the processual approach is intended to improve the quality of the entire research process and, consequently, of the evidence generated, including validity-related actions at all research steps. Thus, our processual validity approach benefits researchers and practitioners by describing validation actions at all steps of the qualitative research process and providing a structured approach to develop better qualitative health care research practices.

Introduction

Society and the scientific community are undergoing significant changes. New technologies, new approaches to non-mainstream problems, and even the revival of old situations (the novel coronavirus [Covid-19] pandemic) have been challenging science to come up with new solutions and more knowledge. In this context, health care and its practices exhibit geographical and social characteristics that must be considered in research design and validation procedures. 1

Although quantitative research approaches in health care are well represented, the potential of qualitative research is still underrepresented or misunderstood. 2 Lack of training in and for qualitative methods, 3 as well as lack of transparency and secure evidence in published qualitative articles, 4 coupled with the deep, intrinsic relationship between qualitative research and the philosophy of science, 5 are some of the difficulties that concern qualitative methods in the field. Although these difficulties are Herculean, the rewards are promising. 6 High-quality qualitative research can lead to insightful contributions to providing secure evidence for decision-making in health care 7 by answering what, how, and why questions about the social aspects of our society’s current health and health care problems. 1

Qualitative research is promising, albeit with new challenges, such as thinking about the criteria needed to obtain high-quality research. 9 Given that there are many different approaches to qualitative studies, the criteria can differ. 10 , 11 However, there is a growing consensus about preoccupation with and adoption of rigorous quality criteria. 12 , 13 The present article does not offer an extreme or “one best way” answer regarding the different qualitative research approaches; rather, it is an attempt to promote dialog and development in the field in order to improve the quality of qualitative research. 14 , 15 This article proposes a flexible and prudent way to ensure quality through the elicitation of different research steps. At each step, there are activities to construct and enhance validity and overall quality. Qualitative researchers should manage their ability to attend diverse activities from the beginning to the end. The processual approach is not a straitjacket to define what activities should be performed because it depends on the nature and purpose of the research project. Several researchers have reported different concepts of quality and rigor, such as trustworthiness and appropriateness, defining them as proxies to highlight the intellectual effort involved in and the seriousness of the research, 16 , 17 or as “accountability” to the scientific community, readers, and the researched people. 18 However, the problem is not the label or the concept per se, but rather the need to adopt a broader view of rigor in qualitative research. However, a major concern regarding qualitative research is the great variety of epistemic, philosophical, and ontological aspects involved. 13 , 14 , 18 Recent evidence suggests that some structured ways of dealing with the diverse quality dimensions of qualitative research, such as internal and external validity, reliability, objectivity, credibility, transferability, dependability, and confirmability, 19 can facilitate the teaching and diffusion of acceptable qualitative research practices. 20

Qualitative validity is also too important and rich to be based on only a few fixed measures at the end of the study, such as Cronbach’s alpha among the quantitative research methods. 21 , 22 Every stage of qualitative research is significant, and a processual approach to ensure validity can influence the quality of all stages. 13 , 14 The process is the heart of our understanding of qualitative research, considering duration and movements, conscience and liberty, knowledge and reflection, experience and interpretation, and researcher and researched matter. 23

Thus, this article aims to explore the application of the processual validity approach, which was developed in the context of the social sciences, to the specific context of health care. 1 , 24 Two healthcare case studies were analyzed to explore the approach’s applicability and the benefits obtained. The benefits are threefold. First, a processual approach can highlight a means to obtain a rigorous, quality research process and secure evidence of the results and findings in qualitative studies. Second, a processual approach may reinforce the importance of qualitative research in the field. For example, Lee et al. 25 used a qualitative method to develop a better understanding of the social and cultural factors affecting Cambodian-American refugees/immigrants and constructed a tailored research instrument for investigating hepatitis B virus (HBV) and human papillomavirus (HPV) prevention behavior. Third, a processual approach is useful to gain a better understanding of the nature and challenges of qualitative research, mainly through opportunities to break with the dualism and separation of researchers and participants, 26 while also promoting systemic reflexivity and a self-conscious research design for or between researchers. 23 , 27

Processual Validity

In qualitative research, validity does not present a unitary concept. 12 , 28 Therefore, a processual approach may offer qualitative research more flexibility to adapt projects to different situations, contexts, epistemological paradigms, and personal styles in conducting research. In addition, qualitative methods should not use very strict or “one best way” strategies. 29 Moreover, in qualitative research, all steps matter. From the beginning of problem construction to the delivery of articles and the final report, preoccupation with and surveillance of validity should be constant. 24 Furthermore, processual validity is not a checklist, 6 , 30 and it can positively influence research toward holism. 31 Processual validity is a methodological construction that involves all the research steps, including those before and after field data collection and analysis.

In other words, the processual operationalization of validity can be planned and implemented in the following sequence: ex-ante field research (Step A), moments of truth (Steps B, C, D), and ex-post field research (Step E). Although the framework is organized linearly, conducting research implies transversal activities such as reflexive analysis and triangulation.

Step A— Ex-ante Field Research

This step includes activities to get acquainted with the research field; establish the research domain, design, and limits; converse with experts and/or visit the field with a broad and open mind; plan the formal research with its protocols and objectives; and develop a pilot test. On the other hand, prolonged engagement, assessing whether the research subject is exciting and innovative, and forecasting theoretical contributions are critical validity criteria. 17 Furthermore, qualitative research is time-intensive; 32 the researcher requires time to go beyond the studied phenomenon’s superficial responses.

Adopting some authors’ 19 definitions of the dimensions of qualitative validity, the ex-ante field research step contributes to assessing the research process’s reliability and transferability.

Steps B, C, and D—Moments of Truth

Step B comprises participant or non-participant data collection using interviews, focal groups, virtual interactions, etc. Step C includes analyzing and codifying the data, structure concepts, and developing categorical evidence and connections. Step D includes activities that support the construction of illations through reflection and reflexivity, the saturation and triangulation of refined data, and temporary knowledge of the phenomenon. Reflection is to “think about it,” 23 whereas reflexivity is an immensely personal and intimate experience that is very close to the researcher. Therefore, prolonged engagement; persistent observation; detailed and rich descriptions of concepts, categories, and structures and processes revealed by the data; explicit separation of the first- and second-order findings; triangulation of data types; interrater reliability; negative case analysis (reflexivity); clarifying researcher bias; peer debriefing; and member checks are important qualitative validity techniques. 17

On the other hand, data integrity, meticulous data management, and recording, that is, verbatim data transcriptions, careful observation notes, clear notes on theoretical and methodological decisions, and accurate records of contacts and interviews, as well as a balance between subjectivity and reflexivity, should be considered as validity techniques involved in qualitative research. 33 In addition, saturation occurs when the researcher continues sampling and analyzing rich and diverse data until no more useful data or information emerges. 34 In other words, even with additional effort and time, there are no further incremental improvements in quality and knowledge if saturation is achieved.

Finally, some authors 35 , 36 have observed the importance of verifying with the interviewees whether the data were correctly understood and receiving feedback on the content of the research report. All tools and techniques contribute to ensuring overall qualitative validity through different steps and activities. 37

Following some authors, 19 important dimensions of the qualitative validity of the research process to assess moments of truth are reliability, objectivity, and transferability of the research process, as well as the credibility of the results obtained.

Step E— Ex-Post Field Research

This step comprises the communication and discussion of the findings and results with other researchers through the presentation of conference papers and submission to peer-reviewed academic journals. Criteria that contribute to the validity of qualitative research are accountability to the scientific community, readers, and the researched people, 17 and the clear communication of findings. 19 In addition, it is important to verify the findings in the “real world” and with other stakeholders, such as the community and journalists, for instance, to obtain insights for generalization and even serendipity. 38 The actions researchers perform at this step contribute decisively to assessing the dependability of the entire research process and the confirmability of the results obtained.

All the steps involved in processual validity matter, and personal and/or group debriefing and reflection are constant. Moreover, the researcher engages in continuous analysis in the field, as the senses capture, select, and analyze everything. 39 In a sense, processual validity implies the importance of verification strategies and the need for preoccupation with different stages of the research process and the researchers. 40 Thus, researchers are essential knowledge construction players in processual validity due to the research field and researchers’ interactions at different times and moments. The process is ongoing, and prudence, transparency, conscience, and attitudes represent essential aspects of qualitative researchers’ skills. 24

Illustrative Cases

To demonstrate the importance and applicability of processual validity, we selected two illustrative cases (Cases 1 and 2) based on the following criteria: knowledge field, health care, and its practices; research articles recently published (2019 or after) in high-quality peer-reviewed journals (impact factor higher than 1.0 or H index higher than 40); access to complete information about the research and methodological choices; research involving different research strategies and designs (i.e., exploratory, descriptive, or explanatory studies, and cross-sectional or longitudinal studies) as well as different methods (i.e., case studies and qualitative inquiries), and research based on different data collection procedures and data analysis methods.

Analysis of Illustrative Cases

Case 1 41 explores the gap between medical education and patients’ needs in primary health care, especially for the most vulnerable population. Strong social inequalities characterize Brazil, and one of the challenges for medical education is to overcome the biomedical paradigm, which is physician- and hospital-centered, and move toward primary health care practices centered on the suffering individual, who is inseparable from their social condition. This investigation used formative evaluation documents, namely field records (i.e., weekly field journals, final reports, and focus group transcripts produced by medical students during a 22- week internship course). To analyze these data, the authors adopted a critical approach grounded in dialectical hermeneutics, cultural–historical activity theory, and the sociocultural theory of learning. In addition, the authors grouped the narratives into five axes. As a result, the research shows how internships reduce the gap between clinical views and user needs.

Case 2 42 presents a qualitative study to explain health care improvements. The authors emphasize the complexity of healthcare organizations. They continuously seek to improve the population’s health and patient experience, while simultaneously attempting to reduce costs and adopt quality improvement strategies. An explanatory case study was conducted at a university hospital. Thirty semi-structured interviews and 250 hours of observations were recorded over 3 years, and 198 reports were gathered. Data were structured and submitted for content analysis based on complexity theory and the adaptive leadership framework. The results showed that quality improvement efforts could benefit from an understanding of the importance of learning and sharing responsibility to deal with the coexisting degrees of contextual complexity in modern health care.

Our Analysis of the Two Cases Followed the Processual Validity Approach

Step A: Ex-ante field research aims to establish the research domain, define the subject of investigation, and recommend previous immersion in the field. At this point, both studies describe previous exposure to the field. Case 1 analyzed data from a course assessment based on the weekly field journals (22) of 55 medical students from two different semesters for a duration of one semester. Four weekly field journals and final reports were selected from this database and analyzed (pre-analysis) to build a preliminary analysis grid. One of the researchers worked as a doctor for several years and developed a critical approach to analyze the data. In Case 2, the authors selected a university hospital to study changes and improve health care. This research was linked to a broader research project at the university. Intensive study of the health sector and in-depth description of the environment and the case facilitate comprehension of the field and the research objective. We found a detailed description of the selected research field and how data were collected in both cases.

The first step of the processual approach aims to foster the researchers’ attention to better understand the object and the studied phenomenon, thus amplifying their perception of the entire context. Contextual information helps the external observer to get a better sense of the “big picture” and, consequently, of the subtle nuances. Detailed information about the context is needed to increase the results’ credibility. 43

Step B concerns data collection. Both cases used a considerable amount of secondary data and several sources to gather data for their investigations. The Case 2 database comprised 22 weekly field journals and final assessment reports produced by 55 students from two different semesters of a medical internship. These reports were developed by medical students and sent to their supervisors. The supervisors returned them with feedback, and final reports were formulated at the end of the internship. Additionally, the researchers used data from focus groups conducted with students at the start and end of the internship to assess their learning experience in contact with patients. In Case 2, data were collected over 3 years. A total of 198 documents (working documents, process maps, presentations, meeting notes, action plans, hospital reports, newspaper articles, and departmental newsletters), 30 semi-structured interviews with people involved with the research problem, and 250 hours of observations conducted in camps, meetings, and workshops composed the data collected. The extensive time the first author spent in the field was integral to collecting this volume of data from different sources, making triangulation feasible.

Step C comprises data codification and analysis. Case 1 used four field reports to perform a pre-analysis and 20 reports––10 field journals and final reports and 10 focal group transcripts (5 from the start and 5 from the end of the course)–to structure the final analysis. Case 2 shows the researchers’ prolonged field exposure (from 2013 to 2018). The first author spent 100 hours in workshops/seminars and 185 hours attending internal meetings. To manage all the data and organize and structure it chronologically and thematically, NVivo software was used.

At Step D, both studies used several data sources to deepen the data analysis by providing multiple detailed descriptions of the phenomena. The adoption of different sources facilitates in-depth analysis and enables researchers to create data and theoretical triangulation. Thus, compiling different pieces of knowledge favors saturation.

Step E consists of ex-post field research and is related to discussing the results with peers and other researchers to complete the theorization process. Researchers should attend scientific events, and peer reviews should be administered to assess the communication of the results. At this step, Case 1 reinforces the findings through one researcher’s practical activities as a family doctor working at the Rio de Janeiro city hall. In addition, an article was produced and presented at the 21 st World Conference on Family Doctors. Case 2 used several sources to validate the analysis and results. This was first done with the head of the studied hospital and the research project’s three co-supervisors (a PhD thesis). One author also attended the ECIC—European Congress on Intrapartum Care—2017 conference as a speaker and Dublin’s EAWOP—European Association of Work and Organizational Psychology—research seminar. This step is particularly important for improving and deepening knowledge because it represents an opportunity to receive criticism, comments, and suggestions, and refine the results.

Based on our literature review, a summary of recommended actions to be taken at the various steps that comprise the processual approach as well as a synthesis of the elements and activities analyzed in the two cases are presented in Table 1 .

Recommended actions and elements and activities analyzed.

It is important to highlight that all steps are recursive. More than standard criteria, qualitative research depends on the researcher’s prudence, transparency, conscience, and attitudes. 24 Ensuring research validity can be laborious; our processual framework seems to help researchers with this relevant task.

Discussion, Implications, and Conclusion

As we pointed out, all the steps involved in processual validity matter, and the proposed integrated approach helps to appropriately link researchers, fields, and data to obtain high-quality results. Both cases illustrate the five processual validity steps, demonstrating the benefits of adopting good research procedures to promote reflexivity and increase the rigor of the study and the quality of the results.

Analyzing the research procedures adopted in Case 1, we first highlight participants’ representativeness, a detailed explanation of the data used, and the researcher’s prolonged exposure to the field. One of the co-authors works as a family doctor. Understanding the field helps the researcher appropriate the “field’s language,” which improves open coding procedures when analyzing the data. The researcher then used double coding techniques (provisional and final analysis grid) and several data sources, which facilitated triangulation and helped engage in reflection and achieve saturation. The adoption of dialectical hermeneutics to analyze the data made it possible to combine interpretive and critical elements, dive deeply into the phenomenon, and promote systematic reflexivity regarding how to make medical students’ internships more effective and more closely aligned with the most vulnerable population’s primary health care needs, thus improving the study’s social relevance. Analyzing the dimensions of validity in Case 1, the transparency of the hermeneutic data analysis process was weak. Missing details compromise reliability and transferability. Rigor in hermeneutic research is much more processual than in methodological research, once the researcher is at the center of the process. 14 In addition, discussion of the results in the ex-post step of the field research has not been detailed, weakening the confirmability of the findings.

In Case 2, the researcher also had long field exposure, but the methodological approach was different, as it was a positivist case study. The data were coded and analyzed using ex-ante analytical categories from two theoretical frameworks and qualitative content analysis. The authors of this article fulfilled all five processual validity steps of the proposed approach, showing high rigor, objectivity, and credibility to secure evidence. A significant difference between studies is related to the delimitation of the five steps identified in Case B and is a consequence of a more objectivist approach.

In a sense, processual validity attempts to create harmony and equilibrium between the “big picture” of the event’s context and the subtle details. In addition, the field and scientific peers can influence approximation, mainly at the last step (E) in the framework. Therefore, a sense of objectivity and intersubjectivity is a condition sine qua non for the quality of qualitative research.

This paper argues for the benefits of adopting a processual approach to validity in health care research. However, both case studies are exploratory, with inherent limitations concerning data and literature. Consequently, it was impossible to present more explanatory conclusions, and further research is required.

Regardless of the qualitative researcher’s epistemological or ontological position, the procedural approach to qualitative studies’ validity proved useful and applicable. According to the proposed framework, we aimed to improve the quality of qualitative research in health care, while helping to clarify some misunderstandings and providing conditions for continuous research development. 9 , 38 The qualitative researcher is not just at the center of the research process; they are also the manager of an equilibrium between the creativity of the arts and the rigidity of science. Hence, every step matters for continuous co-construction. 23 , 37 High-quality qualitative research can yield insightful contributions to providing secure evidence for health care decisions. 39 Thus, in a modified quote, “Validity is not a destination; it is a way of thinking, doing, and living research.”

Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.

Paulo Hayashi Jr https://orcid.org/0000-0003-2718-244

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

Reliability vs. Validity in Research | Difference, Types and Examples

Published on July 3, 2019 by Fiona Middleton . Revised on June 22, 2023.

Reliability and validity are concepts used to evaluate the quality of research. They indicate how well a method , technique. or test measures something. Reliability is about the consistency of a measure, and validity is about the accuracy of a measure.opt

It’s important to consider reliability and validity when you are creating your research design , planning your methods, and writing up your results, especially in quantitative research . Failing to do so can lead to several types of research bias and seriously affect your work.

Table of contents

Understanding reliability vs validity, how are reliability and validity assessed, how to ensure validity and reliability in your research, where to write about reliability and validity in a thesis, other interesting articles.

Reliability and validity are closely related, but they mean different things. A measurement can be reliable without being valid. However, if a measurement is valid, it is usually also reliable.

What is reliability?

Reliability refers to how consistently a method measures something. If the same result can be consistently achieved by using the same methods under the same circumstances, the measurement is considered reliable.

What is validity?

Validity refers to how accurately a method measures what it is intended to measure. If research has high validity, that means it produces results that correspond to real properties, characteristics, and variations in the physical or social world.

High reliability is one indicator that a measurement is valid. If a method is not reliable, it probably isn’t valid.

If the thermometer shows different temperatures each time, even though you have carefully controlled conditions to ensure the sample’s temperature stays the same, the thermometer is probably malfunctioning, and therefore its measurements are not valid.

However, reliability on its own is not enough to ensure validity. Even if a test is reliable, it may not accurately reflect the real situation.

Validity is harder to assess than reliability, but it is even more important. To obtain useful results, the methods you use to collect data must be valid: the research must be measuring what it claims to measure. This ensures that your discussion of the data and the conclusions you draw are also valid.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Reliability can be estimated by comparing different versions of the same measurement. Validity is harder to assess, but it can be estimated by comparing the results to other relevant data or theory. Methods of estimating reliability and validity are usually split up into different types.

Types of reliability

Different types of reliability can be estimated through various statistical methods.

Types of validity

The validity of a measurement can be estimated based on three main types of evidence. Each type can be evaluated through expert judgement or statistical methods.

To assess the validity of a cause-and-effect relationship, you also need to consider internal validity (the design of the experiment ) and external validity (the generalizability of the results).

The reliability and validity of your results depends on creating a strong research design , choosing appropriate methods and samples, and conducting the research carefully and consistently.

Ensuring validity

If you use scores or ratings to measure variations in something (such as psychological traits, levels of ability or physical properties), it’s important that your results reflect the real variations as accurately as possible. Validity should be considered in the very earliest stages of your research, when you decide how you will collect your data.

• Choose appropriate methods of measurement

Ensure that your method and measurement technique are high quality and targeted to measure exactly what you want to know. They should be thoroughly researched and based on existing knowledge.

For example, to collect data on a personality trait, you could use a standardized questionnaire that is considered reliable and valid. If you develop your own questionnaire, it should be based on established theory or findings of previous studies, and the questions should be carefully and precisely worded.

• Use appropriate sampling methods to select your subjects

To produce valid and generalizable results, clearly define the population you are researching (e.g., people from a specific age range, geographical location, or profession).  Ensure that you have enough participants and that they are representative of the population. Failing to do so can lead to sampling bias and selection bias .

Ensuring reliability

Reliability should be considered throughout the data collection process. When you use a tool or technique to collect data, it’s important that the results are precise, stable, and reproducible .

• Apply your methods consistently

Plan your method carefully to make sure you carry out the same steps in the same way for each measurement. This is especially important if multiple researchers are involved.

For example, if you are conducting interviews or observations , clearly define how specific behaviors or responses will be counted, and make sure questions are phrased the same way each time. Failing to do so can lead to errors such as omitted variable bias or information bias .

• Standardize the conditions of your research

When you collect your data, keep the circumstances as consistent as possible to reduce the influence of external factors that might create variation in the results.

For example, in an experimental setup, make sure all participants are given the same information and tested under the same conditions, preferably in a properly randomized setting. Failing to do so can lead to a placebo effect , Hawthorne effect , or other demand characteristics . If participants can guess the aims or objectives of a study, they may attempt to act in more socially desirable ways.

It’s appropriate to discuss reliability and validity in various sections of your thesis or dissertation or research paper . Showing that you have taken them into account in planning your research and interpreting the results makes your work more credible and trustworthy.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Middleton, F. (2023, June 22). Reliability vs. Validity in Research | Difference, Types and Examples. Scribbr. Retrieved September 2, 2024, from https://www.scribbr.com/methodology/reliability-vs-validity/

Is this article helpful?

Fiona Middleton

Other students also liked, what is quantitative research | definition, uses & methods, data collection | definition, methods & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

We Trust in Human Precision

20,000+ Professional Language Experts Ready to Help. Expertise in a variety of Niches.

API Solutions

• API Pricing
• Cost estimate
• Customer loyalty program
• Educational Discount
• Non-Profit Discount
• Green Initiative Discount1

Value-Driven Pricing

Unmatched expertise at affordable rates tailored for your needs. Our services empower you to boost your productivity.

• Special Discounts
• Enterprise transcription solutions
• Enterprise translation solutions
• Transcription/Caption API
• AI Transcription Proofreading API

Trusted by Global Leaders

GoTranscript is the chosen service for top media organizations, universities, and Fortune 50 companies.

GoTranscript

One of the Largest Online Transcription and Translation Agencies in the World. Founded in 2005.

Speaker 1: Validity and reliability are probably among the most confusing and frustrating terms when it comes to qualitative research. There are so many definitions and so many discussions and so many alternative terms have been put forward, so it doesn't really help to understand what validity is and how we can ensure that our findings are valid or how we can increase these findings' validity. So in this video, I'll take you through six steps to increase the validity of your qualitative findings. In quantitative research, validity and reliability are quite straightforward terms. So reliability refers to replicability and consistency of certain measurements and validity to whether this measurement is measuring what it's supposed to measure. So it's quite straightforward. But think about qualitative research. Can we really talk about consistency of our instruments? Imagine that you're interviewing the same person twice and asking the same questions. Even though you're asking the same questions, this person is not likely to give you exactly the same answers. So for this reason, reliability doesn't really refer to qualitative research. It's not that relevant. And usually, people discuss validity rather than reliability of qualitative studies. And validity of qualitative research is usually discussed in terms of three common threads to validity, which are three different types of bias. Respondent bias, researcher bias, and reactivity. So respondent bias refers to a situation where your participants are not giving you honest responses for any reason. They may feel that the topic is threatening to their self-esteem, for example, or they may simply try to please you and give you the answers they think you are looking for. Researcher bias refers to the influence of your previous knowledge and assumptions on your study, which may be a very dangerous and a very risky factor in your study. I've talked about the role of assumptions quite a lot in my other videos and in my blog. And finally, reactivity refers to the role of you as a researcher and your influence, your physical presence in the research situation, and its possible influence on the data, on what the participants say, and so on and so forth. And in order to minimize the potential influence of these three types of bias on your study, Robson suggests the following six strategies to deal with threats to validity. Prolonged involvement refers to you as a researcher being involved in the research situation in your participants' environment, which is likely to result in the increase in the level of trust between you and your participants. This in turn is likely to reduce the risk of respondent bias and reactivity as you generate this common trust. However, it is likely to increase the risk of researcher bias because you and your participants are likely to generate some set of common assumptions. And as I said, assumptions may be a very dangerous thing for your research. Triangulation is such a broad topic and I'm sure that you've at least heard about it before, if not read about it. Triangulation may refer to many things, including triangulation of data, so when you collect different kinds of data, triangulation of methodology, when you have, for example, mixed methods research, or triangulation of theory, where you're comparing what's emerging from your data to previous existing theories. In any case, triangulation is likely to reduce all kinds of threats to validity, so just remember that it's always good to consider triangulating these different aspects of your study. Peer debriefing refers to any input or feedback from other people. This may happen during internal events, such as seminars or workshops in your university, or external, such as conferences. In any case, the feedback and quite likely criticism that you'll receive from other people helps you become more objective and helps you see and become aware of certain limitations of your study. And this is likely to reduce researcher's bias, so again, researcher's bias which was about your previous assumptions and your previous knowledge. So you're becoming more objective and more aware of how your study may be improved. Member checking may mean a couple of things, but in essence it refers to the practice of seeking clarification with your participants. So asking them to clarify certain things before you actually jump into conclusions and describe your interpretation of that data. So it may be simply keeping in touch with your participants, sending them a text message or an email, and asking them whether what you think they meant when they said something in the interview is actually what they meant. Another practice is to send them interview transcripts. So to send them the whole transcript and ask them to delete or change things or add things to that transcript. And finally, you have a method called validation interview, which is all about member checking. So it's basically a whole interview which serves the purpose of this clarification that I discussed. So after you've conducted the first run of analysis after the interview, you conduct another interview and you just ask your participants about your interpretations and about anything that was not clear to you. Negative case analysis is one of my favorite things to do. And I talk extensively about it in my self-study course on how to analyze qualitative data. But basically what it involves is analyzing these cases or data sets that do not match the rest of the data, do not match the trends or patterns that emerge in the rest of the data. And although you may feel tempted to ignore these cases, you may fear that they will ruin your data or your findings, quite often they tell you more about the rest of the data than these actual other cases themselves. So negative cases highlight not just how this one case is different from the rest of the data, but they actually highlight the similarities between the rest of the data. So this is a very, very valuable and important thing to do. And finally, keeping an audit trail means that you keep a record of all the activities involved in your research. So all the audio recordings, your methodological decisions, your researcher diary, your coding book, just having all of this available so you can, for example, demonstrate it to somebody. So again, this way you become really transparent and the validity of your findings cannot really be argued. Importantly, don't worry about having to apply all these strategies in your study. Firstly, some of them are almost natural, like peer debriefing. So as a student, it's very likely that you will receive feedback, you will talk to other people about your study, you will receive feedback and criticism. So you don't really have to worry about consciously applying it as a strategy. And secondly, you can choose some of these strategies, a combination of these strategies. You don't really have to apply every single one on the list. However, it is important to think about validity and it's very important to talk about it in your study. So if you demonstrate that you are thinking about validity and you demonstrate what exactly you did to increase this validity, it will be a major, major advantage to you and to your study.

• Open access
• Published: 02 September 2024

Implementation of health-promoting retail initiatives in the Healthier Choices in Supermarkets Study—qualitative perspectives from a feasibility study

• Katrine Sidenius Duus   ORCID: orcid.org/0000-0002-1630-3132 1 ,
• Tine Tjørnhøj-Thomsen   ORCID: orcid.org/0000-0003-3621-6682 1 &
• Rikke Fredenslund Krølner   ORCID: orcid.org/0000-0002-4928-4310 1  

BMC Medicine volume  22 , Article number:  349 ( 2024 ) Cite this article

Metrics details

Improving food environments like supermarkets has the potential to affect customers’ health positively. Scholars suggest researchers and retailers collaborate closely on implementing and testing such health-promoting interventions, but knowledge of the implementation of such interventions is limited. We explore the implementation of four health-promoting food retail initiatives selected and developed by a partnership between a research institution, a large retail group, and a non-governmental organisation.

The four initiatives included downsizing of bags for pick’n’ mix sweets and soda bottles at the check-out registers, shelf tags promoting healthier breakfast cereal options, and replacing a complimentary bun with a banana offered to children. The initiatives were implemented for 6 weeks (or longer if the store manager allowed it) in one store in Copenhagen, Denmark. Data were collected through observations, informal interviews with customers, and semi-structured interviews with retailers. We conducted a thematic analysis of transcripts and field notes inspired by process evaluation concepts and included quantitative summaries of selected data.

Two out of four initiatives were not implemented as intended. The implementation was delayed due to delivery issues, which also resulted in soda bottles not being downsized as intended. The maintenance of the shelf tags decreased over time. Retailers expressed different levels of acceptability towards the initiatives, with a preference for the complimentary banana for children. This was also the only initiative noticed by customers with both positive and negative responses. Barriers and facilitators of implementation fell into three themes: Health is not the number one priority, general capacity of retailers, and influence of customers and other stakeholders on store operation.

Conclusions

The retailers’ interests, priorities, and general capacity influenced the initiative implementation. Retailers’ acceptability of the initiatives was mixed despite their involvement in the pre-intervention phase. Our study also suggests that customer responses towards health-promoting initiatives, as well as cooperation with suppliers and manufacturers in the development phase, may be determining to successful implementation. Future studies should explore strategies to facilitate implementation, which can be applied prior to and during the intervention.

Peer Review reports

What we eat affects our health and well-being [ 1 ]. Diet is associated with obesity, cancers [ 2 ], and mental well-being [ 3 ], and a healthy diet has been associated with lower all-cause mortality [ 4 ]. One important factor in improving diet is to create a food environment that supports a healthy diet [ 5 , 6 ]. In modern societies, such as Denmark, supermarkets are the main source of food [ 7 ]. Supermarkets therefore hold a significant influence on what food we buy and potentially also eat [ 7 , 8 , 9 ]. Studies report associations between the concentration of supermarkets and overweight and obesity in the neighbourhood [ 10 ] and between the healthfulness of supermarkets and people’s diets [ 11 , 12 ]. Moreover, unhealthy food and beverage products are promoted more often than healthy products and beverages in, for example, supermarkets [ 9 , 13 , 14 ]. This indicates a need to explore how and if it is possible to implement health promotion initiatives in supermarkets and whether customers respond to such initiatives as intended.

Studies show that health-promoting interventions in supermarkets can affect customers to purchase more healthy products [ 7 , 9 , 15 , 16 , 17 ]. Reviews and a meta-analysis have concluded that the most effective initiative in supermarket settings is price changes—the evidence points to the positive effects of reduced prices to increase the purchase of healthier products, especially fruit and vegetables [ 7 , 17 ]. Even though price reductions seem to be effective, they seem more challenging to implement due to retailers’ drive for profit and low preference for financing such price cuts [ 7 , 18 ]. There is some evidence that nudges in terms of product information and positioning, as well as altering the number of available products, can impact what products are being purchased [ 15 , 16 ]. However, the quality of this evidence is low. Overall, most of the studies that have explored the effect of interventions in supermarkets have been conducted in the USA and other high-income countries [ 15 , 16 ], in controlled settings, or applied a weak study design, such as non-randomised studies [ 16 , 17 ]. To our knowledge, only a few studies have been conducted in Denmark [ 19 , 20 , 21 , 22 , 23 , 24 , 25 ]. These studies represent different designs and types of interventions: reformulation of private-label products to reduce calorie content [ 24 ], informational claims to promote low-salt foods [ 23 ], nudges via signs to promote sales of fruit and vegetables [ 22 ], positioning (shelf-space management) of dairy products [ 20 ], replacement of sugar confectionery with fruit and healthy snacks at the checkout [ 19 ], discount on fruit and vegetables combined with space management [ 25 ] and structural changes in supermarkets and education of supermarket employees as part of a multicomponent intervention [ 21 ] (the three latter studies are reporting from the same project). All but one study [ 23 ] found an effect of the applied intervention strategies, although mostly small or modest. This calls for more studies in real-life settings and investigations of why some interventions have the desired effect while others do not. Lack of effect may be explained by 1) customers not noticing or finding the initiatives relevant [ 19 , 23 ], 2) customers buying other products instead of or additionally to promoted intervention products [ 20 , 24 ], 3) the shelf organising effect [ 20 ], or 4) theory fail in regards to customer behaviour [ 22 ].

Several studies have explored facilitators and barriers to the implementation of health-promoting interventions in supermarkets. Reviews show that implementation is supported if the retailer is receptive to innovation, feels responsible for community health, and receives financial support or subsidies [ 26 ]. Furthermore, implementation is supported if the intervention provides the retailers with knowledge of health promotion and business skills [ 26 , 27 ]. Other facilitators include compatibility with context and customers’ needs, positive customer responses to the initiative, the prospect of improved public image, establishment of partnerships, low retailer effort requirements, and increased profit or sales [ 26 , 27 ]. Health-promoting interventions in supermarkets are hindered by high customer demand for unhealthy products and lower demand for healthy products, constraints of store infrastructure, challenges in product supply, high staff turnover, and lack of time [ 26 , 27 ]. Other barriers are doubt regarding changing customers’ behaviour, poor communication between collaborators [ 26 ], high running costs, and risk of spoilage [ 26 , 27 ].

Middle et al. [ 26 ] conclude that the underlying mechanism of barriers and facilitators of implementation is the (mis)alignment of retailers’ and intervention researchers’ interests. The authors, therefore, suggest a close collaboration between intervention researchers and retailers to work towards an alignment of interests and resolving or avoiding misalignment, which is supported by Gupta et al. [ 27 ]. However, knowledge of how such collaborative efforts affect the implementation of healthy food retail interventions is warranted.

The aim of this study is to explore the implementation, acceptability, and feasibility of four different health-promoting food retail initiatives to increase customers’ purchase of healthy food and beverages, which were selected and developed together with food retailers: 1) Promotion of healthier breakfast cereals and products using shelf tags, 2) downsizing of sodas sold at the checkout desks, 3) downsizing of bags for the pick’n’ mix sweets, 4) replacement of a complimentary bun for children with a banana. The study has three research objectives:

To document the implementation and sustainment of the initiatives over time

To explore the retailers’ and customers’ responses to and acceptability of the initiatives

To investigate barriers and facilitators of implementation and sustainment of the initiatives.

Setting and the initiatives

This study was conducted in Denmark during 2020 and 2021, 2 years that involved two major societal events, first the coronavirus disease pandemic and later the start of the Russia-Ukraine war. Both events heavily influenced the circumstances of everyday life including opportunities for conducting research and running businesses. The specific influences on this study will be unfolded later in the findings and discussion sections.

In this study, we collaborated with the retailer Salling Group, which holds 34.2% of the market share of grocery retailers in Denmark [ 28 ]. Salling Group is owned by the Salling Foundations and has no shareholders—all profits go to reinvestment in the business and donations to sports (amateur and professional), charity, education, and research. Salling Group owns three national supermarket chains: føtex, Netto and Bilka, alongside other businesses. For the feasibility test, we collaborated with føtex, which owns over 100 stores all over Denmark, including 23 stores called føtex food. føtex (except føtex food) offers both groceries and many different non-food products (e.g. textiles, cosmetics, toys, electronics, and home accessories).

The initiatives were selected and developed by a partnership, including a group of researchers at the National Institute of Public Health, University of Southern Denmark, consultants from the Danish Cancer Society, and employees at the Corporate Social Responsibility (CSR) department in Salling Group, the marketing department at føtex, and two store managers (hereafter referred collectively to as ‘the retailers’) over approximately 2 years. The process involved in-person meetings, desk research (the use of existing material [ 29 ]), visits to the test store, and a prototype test of three suggested initiatives. The researchers initiated the collaboration and were responsible for designing the research study and data collection and analyses. The retailers hosted the site of the feasibility test, contributed to the selection and development of initiatives and co-managed the practical part of the study. The Danish Cancer Society was recruited by the research project to develop the initiatives. A detailed description of the collaboration and development process is reported elsewhere (Duus et al.  unpublished ).

The feasibility test ended up including four initiatives: 1) Promotion of healthier breakfast cereals and products using shelf tags, 2) downsizing of soda sold at the checkout desks, 3) downsizing of bags for the pick’n’ mix sweets, 4) replacement of a complimentary bun for children with a banana (suggested by the retailers). The initiatives were based on a compromise between the willingness of the retailers and the interest and ideas of the remaining partners rather than on what the literature suggests are the most effective strategies (Duus et al.  unpublished ). Detailed descriptions of the initiatives and the rationale behind them are found in Table 1 .

The prototype test showed that 1) It was important to have a sign informing the customers about the initiative that offered a free banana to children instead of the usual free bun to create a better understanding of the changed offer; 2) Promotional shelf tags needed weekly maintenance as some would fall off; 3) It was difficult to sustain an initiative promoting ready-to-serve salads and ready-to-cook vegetables next to different fresh meats, as it met resistance among the staff due to being an additional task and led to more product waste (Customers did not expect to find these products next to the meat and therefore might not notice them). The learnings from the prototype test led to modifications of the implementation plan and the discard of the latter initiative. The prototype test also made us aware of how quickly the selection of food offered and the layout of the store changed over time, which the researcher, therefore, paid extra attention to during subsequent data collection. Moreover, the researcher made sure to update the list of products that should have a shelf tag a few weeks before the implementation to include new products offered.

The føtex marketing department developed a script to inform the staff at the test store about the feasibility test, explaining and showing each initiative and the aim of the study overall. This was sent to the store manager after being reviewed by the researchers. The store manager was responsible for informing all relevant staff about the implementation and maintenance of the initiatives. The føtex marketing department also made sure to inform the relevant suppliers. Employees at the test store and brand staff from a brewery (who stock the coolers at the check-out desks) implemented the initiatives in the store. The research group did not correct or maintain the initiatives in the store after they were launched; however, the researchers monitored it and reported back to the retailers, either at meetings or by email.

Overall study design

The four initiatives were implemented in the test store for 6 weeks (or longer if the store manager allowed it) starting in September 2021. A føtex store in central Copenhagen (the capital city of Denmark) was chosen as the test store. This decision was made for pragmatic reasons, as the research institute is based in Copenhagen, and based on Salling Group’s decision as it offered their new store layout, which all stores were in the process of being converted to (it was the same store as where the prototype test was conducted).

We designed a qualitative study involving participant observations and interviews to evaluate the feasibility of the initiatives. The methods were designed to explore the partnership and collaboration (the aim of another publication [Duus et al. Unpublished ]), as well as the implementation of the initiatives [ 30 ]. In the design of this study, we were inspired by McGill et al.'s (2020) two-phase framework of qualitative process evaluation from a complex systems perspective. This framework suggests an evaluation that looks at changes over time, starting with phase 1, the static system description and hypothesis generation about how the system might change when the intervention is introduced, followed by phase 2, an adaptive evaluation approach to the system undergoing change which follows emerging findings [ 31 ].

Data collection

In-store observations.

During October and November 2020, we mapped the store layout and customer flow in the test store as part of the static system description. Over 3 weeks, three research assistants performed 12 participant observations of 1005 min in total. The observations followed an observation guide which covered 1) the physical setting (e.g. the layout, placement of products, signs, and pictures); 2) the people (e.g. who are the customers? Are people shopping alone or together with others? How do they move around the store? What are the staff doing?) and 3) short interviews with customers (if possible) about their shopping at the particular store, and their thoughts about the layout of the store. The research teams’ access to the store was approved by the store manager, and research assistants wore a key chain with a sign showing their name and affiliation during the observations. During this data collection period, it was made mandatory to use face masks in supermarkets due to the coronavirus disease pandemic. As the implementation was delayed to approximately 1 year after this static description was completed, one participant observation in the test store was performed at the end of August 2021, just before initiative implementation, to document any major changes in the store layout and selection. Key lessons from these observations about the test supermarket and customers’ behaviour in the store included knowledge on 1) the route around the store, 2) the different times spent at the store, 3) interactions with objects (e.g. products and phones), 4) interactions with children, 5) behaviour of the staff, and 6) sensory impression (Additional file 1). These lessons informed our following data generation and assisted in contextualising our analysis.

The first author monitored the implementation process through participant observations of status meetings ( n  = 2) and correspondence via email and phone with the store manager and the contact person at føtex. In-store participant observations were conducted during and after the feasibility test period, September 2021–May 2022 ( n  = 25 ~ 1795 min in total; see Additional file 2). These observations focused on documenting the presence of the initiatives as well as customers’ and staff’s responses to the initiatives. Access to the store was once again approved by the store manager, and the researcher wore a key chain. During the participant observations in-store, we conducted informal interviews with customers (see Additional file 2 for examples of questions), which lasted a maximum of 5 min each. The first author would approach people and ask if they were interested in answering a brief question. She introduced herself by her first name, where she worked and explained she was doing a research project about shopping patterns. The participant observations were documented by taking notes and photos. Handwritten notes were digitalised and written down at the first chance after leaving the store.

Qualitative interviews

Between November 2021 and February 2023, the first author conducted four semi-structured interviews with retailers ( n  = 3) who had been involved in the study (Table 2 ) to explore their views on the initiatives and the implementation process. Interview guides were used in all interviews alongside different prompts (e.g. timelines and documents). Interview guides were tailored to each participant’s specific role and involvement in the development and implementation of the initiatives. Besides questions related to the initiatives and the implementation effort, the guides included questions about the informants’ background and motivation for the project (personally and professionally), their view on their role and scope for action (individually and organisationally) and their perception of the collaboration with the other organisations. After the participants’ consent was given verbally right before the interview, the interviews were recorded and later transcribed verbatim.

To explore the level of implementation (research objective I), all field notes and photos taken during and after the feasibility test were reviewed to assess whether the initiatives were present and to what degree (e.g. x out of x possible tags).

To explore the perception of the initiatives among employees and customers (research objective II) and identify barriers and facilitators for implementing the initiatives (research objective III), we followed a thematic analysis inspired by Braun and Clarke [ 32 ]. Firstly, field notes and interview transcripts were read thoroughly and openly coded, by writing keywords in the margin of the material, with a focus on the two research objectives. After initial coding, the codes were summarised into broader themes, by writing them into a document with short descriptions and revised according to data excerpts and the full empirical material. The themes drew on the process evaluation concepts: acceptability, responsiveness [ 30 ], motivation, general capacity to implement [ 33 ] and commercial viability [ 34 ]. Lastly, the themes were named, and the final analysis was written up.

We have structured the presentation of study findings as follows: Firstly, we present the implementation of the initiatives overall. Secondly, we present the implementation of each initiative, customers’ responses to them, and the retailers’ perspectives. Lastly, we present the overall facilitators and barriers to the implementation of the initiatives.

Implementation of the initiatives

The implementation of the initiatives was challenged. Firstly, we found that not all the preparations for the implementation were finished in time for the scheduled day. On the scheduled day, the retailer decided to push back the implementation by 1 week. The main reasons were that there had been some misunderstandings around the ordering of the smaller sodas. It was informed that the smaller soda would be a 330 ml can instead of the 375 ml bottle at the price of DKK 10.00 (~ 1.3 euros). The 500 ml bottle usually sold at the coolers cost DKK 16.00 (~ 2.2 euros). The Danish Cancer Society and the research group had two concerns about this: 1) the use of a can instead of a bottle would make the interpretation of the results very difficult, as the bottle and the can have two different functions to the customer—with the can, the product would be consumed all at once, whereas the bottle with the screw lid could be saved for later after it had been open; 2) the price was too low—the price per litre would be lower on the smaller sodas than it had been on those replaced. No changes were made despite these concerns.

Secondly, just days before the implementation, the retailers informed the other partners that they would stick with cans for the test of smaller-sized sodas and that they would now be 250 ml. They acknowledged that both the size and the packing were not optimal but that the optimal 375 ml in a bottle was just not possible. Additionally, they informed the researchers that they could no longer find the new bags produced for the pick’n’mix sweet display.

These challenges led to a delay of the implementation of the initiatives by 1 week, but also a staggered implementation, where the initiatives were implemented when ready (the soda initiative 2 weeks later and the bags for pick’n’ mix sweets 8 weeks later). The retailers agreed to push back the end day correspondingly, upholding the 6 weeks of implementation. Table 3 shows an overview of the implementation of the four initiatives according to the day and week of the feasibility test period.

Smaller product sizes of sodas at the checkout desk

As seen from Table  3 , we did observe the implementation of a smaller product size of the targeted sodas in all coolers, besides the one at the bakery, in the week leading up to the agreed date. We hereafter observed a full implementation of 250 ml cans during the first 2 weeks of implementation. During the third week and the beginning of the fourth week, we observed a mix of 250 and 330 ml cans or only 330 ml cans. The store manager explained that this was probably due to non-delivering from the supplier. At the end of the fourth week and for the last 2 weeks, we observed a full implementation of 250 ml cans. As the targeted size of the initiative was a 375 ml bottle, the initiative was not implemented as intended. After the 6-week feasibility test period, we observed that the smaller 250 ml cans were available in all coolers for at least eight more weeks. As expected, the presentation of the coolers fluctuated over the period. On days of stocking (Monday, Wednesday, and Friday), the coolers would look neat and full, while they would appear more empty or messy on other days.

Customer responsiveness

We observed very few customers who bought any products from the coolers, and we did not get to talk to any customers about the initiative. However, the observations in the store showed no distinct change in customers’ behaviour around the coolers nor expressions of discontent or excitement with the initiative. In an interview with the store manager, he explained that he believed customers had not noticed the change.

Retailer perspectives

The store manager was positive about the initiative, but from his perspective, the decision to implement it should be made at the procurement level and by the suppliers. However, he did have an opinion on how to implement it. The price needed to be fair according to the product it replaced. Moreover, he drew attention to the fact that it was the supplier’s personnel who stocked the products rather than his own. The store manager was, therefore, not surprised that the employees at the store had little to say about the initiative. føtex’s representative (B) was also positive about the initiative and expressed in the interview that the chain would be willing to implement it—if they found it to be the ‘right thing’ to do. However, the representative also emphasised the importance of agreeing with the suppliers, which is a time-consuming process and ‘not done in just six months’.

Shelf tags for breakfast cereal products

From the first day of the implementation, some tags were missing, and one tag was consistently misplaced (Table  3 ). During the first 3 weeks, 10% ( n  = 3) of the tags were missing. This portion progressively increased to 23% until the end of the fifth week. In the sixth week, the portion decreased at first to 16% but decreased again and ended at 26%. In the weeks after the implementation period, the tags stayed present but slowly came off. Approximately 6 months later, three (10%) of the tags were still present. We observed throughout the feasibility test that the presentation of the area varied, which is to be expected in a busy supermarket. At times, the area looked messy; boxes would block access to some products, products would be sold out, some would change packaging, and new products would be introduced to the selection.

When we asked customers about the tags, we learned that they had been unaware of them and that some believed that it was not something they would use—some did not know the meaning of the labels on the tags, while others did not find the labels relevant for them.

[The tags] don’t matter. My wife is pretty health conscious, so we don’t use those, let alone know with such a thing as breakfast cereal. (Male customer)

From our observations of the behaviour of the customers in the breakfast products and cereals department, we find two interesting groups: Those who shop alone and those who shop together with others (primarily children). These groups seem to practice different behaviours.

Among those who do their grocery shopping by themselves, we find two subgroups: 1) those who have planned or know exactly what they want to buy, and 2) those who decide at the store. For the first sub-group, we observed that some showed this by practising a behaviour where they would walk quickly and purposefully towards the shelves and quickly pick up a product. Others would look determined to find a specific product, as the fieldnote excerpt illustrates:

A woman stands looking at the muesli. She first grabs an orange bag on the bottom shelf, then a more yellow one next door and puts the first one back on the shelf. She inspects the bag she took. She starts to look around the shelves more and reaches for a bag that has a pinker look on the top shelf. She puts it back and reaches into the space next to it, where there are a few bags at the very back, but she has difficulty reaching them. A man comes by, notices the woman, and offers to help her. The woman indicates a yes, and the man reaches up and grabs a bag ‘that's the one!’ says the woman as the man hands her the bag.

Another example was a man who kept looking back and forth between some muesli and granola products and his phone before he eventually chose a product. It is unknown whether the man was looking at a specific note, a text request from his family, or a picture on his phone, yet what was on his phone seemed to determine the product he bought. Overall, this group seemed very unlikely to be influenced by the tags, as they had made their choice already before they entered the store.

For the second sub-group, those who seemed to make their decision in the store, we observed that some would just stop and glance at the products without choosing one before moving on with their shopping. Others would look more randomly at the selection than those described above, walk back and forth in the aisle, compare different products and read the info on the back of the products.

For those who shopped together with others (most often children), we observed that when adults shopped with children, the choices of the child and the choices of the adult often conflicted. In one example of a child and a woman who looked at breakfast cereal products, the child was initially allowed to pick a product and asked for different chocolate variants, which all featured cartoon figures; however, the woman rejected all of the child’s choices. In the interaction, the child was met with demands from the woman regarding the attributes of the products: they could not contain chocolate or sugar. In the end, it was the woman who chose a product based on her experience of the child’s preferences and her criteria. In similar situations, we did observe an attempt at compromising between the adult’s and the child’s criteria, which was explained by this woman:

I ask them [woman and boy aged about 10] what they look for when choosing breakfast cereals. The woman looks at the boy and says, ‘Well, what are we looking for?’. The boy does not answer but looks at her and me and smiles. The woman herself replies, ‘Something we can agree on. Something he likes but is not too unhealthy, either’. I ask her what she considers unhealthy. She waffles for a bit and then replies, ‘Yes, but he wants that Lions cereal, for example, and I don’t want him to have that. So something that’s not de facto sweets’. She takes the box of granola that they have chosen [Paulún's blueberry/lemon granola] out of the basket, looks at it and says, ‘So we chose this one. There's probably also a lot of fructose and caramelised stuff in it, but yeah.’

This illustrates the high impact children had on the choices of breakfast products, but also how the parents tried to control and negotiate the final choice.

Retailer perspective

The store manager had little faith in the effectiveness of the shelf tags:

The thing about tagging cereals, I don't think that makes the slightest difference. The reason why I’m sceptical in that regard is that it’s a mixture of what I do on a daily basis. It’s especially the behavioural patterns of our customers, but also how I act as a customer myself to a degree. I don't think shelf tags with the whole grain label or anything like that; in my experience it hasn’t changed things much. (Store manager)

His view on the effect of the initiative was in line with our observations of the customers in the store. Furthermore, the store manager explained that it was difficult to maintain the initiative, as it was not part of the employees’ daily routine. This was also the argument of why the tags lingered after the test period—it was just not part of the usual protocol either to hang them up or take them down. This perspective was shared by the føtex representative (B), who also highlighted the cost of this maintenance.

Contrary to the store managers’ sceptics, the føtex representative (B) was more positive about the initiative:

I think it’s a good initiative. We work a lot with tags and labels in general. [...] I think making it transparent to the consumer is really interesting because there’s nothing wrong with buying a box of Nesquick cereal every once in a while. At least we should not claim it’s the wrong thing to do. But you just have to be clear about what you’re buying, and I think those labels help with that. (føtex representative (B))

She explained that the initiative was highly compatible with their usual strategies. However, she also explained in the interview that a barrier to using shelf tags to promote the buying of certain products was that the chain was trying to reduce the printed material they used in their stores as part of their CSR strategy and to reduce costs.

Replacement of the complimentary bun for children with a banana

The complimentary banana was fully implemented in the feasibility test period except for 1 day of observation, where the signs were not visible (Table  3 ). The initiative also remained available and present by the sign for at least 10 weeks after the implementation period. Furthermore, the store manager informed the researcher that they would continue to provide bananas for customers requesting this as an act of customer service. From the observations, we do find that the presentation of the initiative changed throughout the period. At first, the bananas were placed in a cardboard box on the display counter, which was later replaced with a nicer-looking basket. The number of bananas and their colour also fluctuated during the different days, which would be expected due to the delivery of the bananas and how often they are restocked. However, compared to the buns, we never observed that the bananas were not available, making it a reliable offer no matter the time of the day.

We observed two ways (1 and 2) that the complimentary offer for children was brought up: 1) A customer would ask for the ‘bun for children’, or 2) the staff would offer the complimentary banana to buying customers. In the first way 1), we saw two responses from the staff (a and b) and the customers (i and ii): (a) The customer would be offered the bun with no mention of the banana, or (b) the staff would inform the customer that they no longer offered buns but that they offered a banana instead. The customers had two primary responses to this message: (i) The customer rejected the offer and decided to buy a bun or another item instead. The child was often included in this decision. (ii) The customer accepted the offer and received the banana. In some cases, the child did not accept the offer and the customer compensated for this response by buying a bun or another product for the child. In the second way 2), in which the staff offered the banana spontaneously, the customers almost always reacted positively and accepted the offer.

The following excerpt illustrates why some customers rejected the offer:

A woman with a child of about 1-year-old in a stroller walks up to the bakery and asks for a children's bun. The child has already noticed the buns from the moment they arrive and sits, pointing at the buns through the glass window and babbling. The shop assistant says that there are no children's buns but bananas and points to the sign. The woman replies, ‘I’d like to buy a bun, then’. The assistant takes the bun and enters it into the till, while the woman says, ‘Bananas are so messy’. The assistant smiles and says, ‘Well yeah, I'll pass that on’. The woman replies, ‘It's just that the banana is rather a bother, and the assistant replies, ‘But I think we’ll be offering [the buns] again eventually’.

Thus, adults rejected the offer because eating a banana was a messier process than eating a bun. During meetings and interviews, the retailer also highlighted this as the main reason for rejections of the offer, especially among those with younger children. Another reason for rejection was that the parents did not appreciate the offer nor perceived a need to offer their children a banana instead of a bun.

This initiative was the most successful and interesting one in the eyes of the store manager.

I’d like to highlight the banana for kids, which is clearly the initiative I found most customers were pleased with. (Store manager)

Many customers responded positively to the new offer, which was emphasised as a marker of success. It was also the reason why the initiative continued after the 6-week period, and the store manager explained that they would continue to give bananas to those who asked for them.

The following excerpt illustrates what the bun meant to føtex and the chain’s relationship with its customers.

The children's bun has been around for donkey’s years, and it’s become ingrained in parents and kids alike that you can get them in føtex. So, we’re quite interested in learning how many people would actually, if presented with the alternative, choose something else, like, for example, the banana. I’m quite surprised by that – we can't track it, unfortunately – but off the top of my head, up to 40 to 50 percent actually choose the banana. I find that very interesting. (føtex representative (B))

Thus, it came as a surprise that the initiative was so well received. However, despite the positive experiences with the initiative, the retailers also commented on the cost. They highlighted that the banana was more expensive than the bun, and if it should be an option offered in all stores, then it would have to be prioritised at the executive level as an additional expenditure. In this case, the banana would only be an alternative to the bun and not a replacement. This was rationalised by the retailers’ attitude of not making choices on behalf of the customers.

Smaller bags for pick’n’ mix sweets

This initiative was not implemented until 8 weeks after the initial implementation date. It was fully implemented for five out of the six weeks; during the third week, we observed that the old, larger bags had been hung in front of the new smaller bags. At 2 weeks and four and a half months after the feasibility test, the smaller bags could still be found behind the larger bags—however, it is unlikely that these would have been used, as the obvious choice would have been the bag at the very front. As described for the other areas, this also fluctuated in its presentation and stocking.

We did not get any direct reactions from customers on the smaller bag. However, our observations showed that different strategies were used to decide the amount of candy among customers who bought pick’n’mix sweets. Some showed signs of visually assessing the amount of sweets in the bag, which were the customers we would expect to influence. We often observed this strategy among adults with children, where it was the adult who would visually assess the amount and communicate to the child when they had picked enough.

Those with very young children would walk alongside the child and select the sweets for them, and some adults would encourage the choice of the child by pointing out different variants and commenting on the appearance of the sweets.

Other strategies were to mix according to a pre-defined number of pieces or volume:

A boy of about 10 and a girl of about 8 come over and mix sweets. They repeatedly weigh the bag while doing so. A woman comes over, and the girl says, ‘Hello mummy!’ The woman says, ‘Don’t forget to weigh it’. She then grabs a bag herself and begins to mix sweets. The boy asks the girl, ‘Did you weigh it?’. The girl walks over to the scales and says, ‘I think I’ve got enough’. However, she does not close the bag, and she begins to walk around somewhat restlessly, then says, ‘I don’t know what to pick. I’m still [a few] grammes short’.

An interesting aspect of the situation above is that the girl expressed that she was satisfied with what she had chosen, but she felt that she had to meet the prespecified weight and, therefore, tried to find more sweets to put in her bag. Such strategies undermine the mechanism which the initiative was trying to influence.

Overall, the retailers were positive about this initiative. The føtex representative (B) highlighted that this initiative was interesting as it was a stealth initiative, compared to the initiatives with the sodas, and would change the behaviour of the customers without them noticing. In her opinion, this was not a problem, as people paid per gram.

The store manager had a clear demand for the implementation; it should be easy for both the staff and customers to use. This perspective was backed up by a føtex representative (B) who said:

If there’s something that doesn’t work for us, it’s... if it doesn’t work for our customers, that’s what we need to solve first. (føtex representative (B))

This shows how one success criterion of the retailers is customer satisfaction, which we elaborate on later (See: Influence of customers and other stakeholders on store operation).

The initiative was very delayed, and one reason was that it was challenging to create a new bag that would work in the store. This resulted in the order of many different bags in large quantities due to the agreements with the suppliers, which had been very costly for the retailer.

The føtex representative (B) also reflected on what the potential evidence of an effect would mean to the retailer:

Then we’ll have to wait and see if people buy fewer sweets. And of course, this is something that we must take into account because it’s no secret that part of being a responsible business is to make a profit. And if we sell fewer sweets, then we make less money. (føtex representative (B))

This shows how health and financial profit were seen as opposites and how the success of the initiative would not necessarily lead to it being viewed favourably, as it would negatively affect their profit. Any implementation in the chain would, therefore, have to be a strategic decision.

Facilitators and barriers

In the sections above, we have focused on the four specific initiatives. In the following, we will present analytical findings that go across the initiatives and elucidate what facilitated and hampered the implementation of the initiatives overall. We have organised our findings under three headings: Health is not the number one priority; General capacity of the retailer; and Influence of customers and other stakeholders on store operation.

Health is not the number one priority

In this section, we present the retailers’ motivation for and interest in engaging in the project and working with health and health promotion and what drives and/or curbs this motivation. In our understanding of motivation, we draw on Scaccia et al. [ 33 ] and view motivation as incentives and disincentives that contribute to the desirability of using an initiative focusing on health.

We find that the retailers expressed motivation for working with health and health promotion, which at first seemed to be based on interest. The retailer representatives explained how they personally were interested in health and wanted to learn more, but also that the organisation had an interest in health, especially among children and young people, and wanted to contribute to health-preventing activities, for example, by financially supporting local sports clubs. According to one retailer representative, this was because physical activity and healthy eating promote happier customers, as well as happy employees. The argument points to retailers’ focus on customer satisfaction (see: Influence of customers and other stakeholders on store operation). The focus on the customers relates to another factor of motivation: Working with health was also seen as a relative advantage in that customers increasingly demand healthier products and alternatives. Lastly, we found that the motivation for working with health was a feeling of obligation due to the view of having a social responsibility:

I would say, in purely business and commercial terms, we are, indeed, a commercial business that was created to make money. There’s no ignoring that (laughs). So, of course, this is our main KPI [key performance indicator]. But that being said, we also agree that we have a social responsibility because we are as big as we are. We make a lot of foodstuffs available to the Danes, as do many of our colleagues in our industry, so there is no doubt that we have a role to play in terms of what we make available. (føtex representative (A))

According to the excerpt, this obligation was rooted in the size of the organisation and, thereby, the major influence on people’s selection of food products. However, the excerpt also highlighted that health was not their first priority, which was profit. This point has been repeatedly mentioned among retailers, which reinforces its validity; they were a business and had to gain profit to keep running their operation, which presented limits for what could be implemented. The store manager even expressed how he perceived the running of a supermarket and promotion of public health as incompatible goals and something he had never seen an example of in a real-life supermarket.

However, from the interviews with the retailers and our fieldwork, it seemed that this was not completely black and white, as the retailers were willing to give up their profit in some cases. An example is the hiding of tobacco products in all Salling Groups’ supermarket chains, which they voluntarily implemented in 2018, which led to a significant decrease in profit from tobacco products.

After all, the Salling Group pioneered this with tobacco products. I'm proud of that, but I also think it’s the right thing to do. My personal opinion is that it was the absolutely correct move they chose to make, by making it harder to market a product that is obviously bad for my health. We’re not there with pick‘n’mix sweets just yet, in that we would claim they’re bad for your health, but the mindset in terms of; that is, upholding the mindset when it comes to cigarettes is something that we, as an industry, can easily support in close cooperation with, among others, yourselves [researchers] and the industry. (Store manager)

Risk seemed to be the driver. If the retailer was convinced that the risk was real or big enough, then they were willing to give up some of their profits because it was the ‘right thing to do’, and they would have the courage and power to do so. It was mentioned by all three informants that they did not believe in bans, limitations or hiding of products, as this interfered with the customer’s freedom of choice. This viewpoint was a barrier to the implementation of all initiatives that used strategies that would minimise or reduce the availability of a product. Yet, as with the tobacco products, we found other examples where this restriction of choice was justified by the retailer. One example was that the føtex chain only sold organic bananas. From a sign in the store, this was because:

‘we want to avoid the spray agent chlorpyrifos. Among other things, it is suspected of harming the development of children and foetuses. We can’t live with that suspicion and therefore you can only buy organic bananas in the future’

As with the cigarettes, the argument here was the health risks. In the interview with the store manager about restricting choices, animal welfare and political reasons (e.g. Russia’s warfare against Ukraine) were mentioned as other arguments for doing so.

So, despite an immediate motivation for working with health, the retailer also expressed how other interests and priorities may hinder and set aside the work with health.

General capacity of the retailer

This section presents our findings relating to the general capacity of the retailer in the form of resources, organisational size, and culture. General capacity is understood as the readiness or ability to implement any new initiative [ 33 ].

Through the interviews with the føtex representative (B) and working together with the retailer during the project, we have found that the retailer seemed to be used to and willing to implement new initiatives. In this current study, they accounted for all expenses related to the development of materials for the test and were also willing to risk some of their profit for a short period of time. The føtex representative (B) highlighted this high level of available resources several times in the interview:

I have some leverage, so when we do something, we don’t do it by halves. What I find most motivating, and I can say that with complete peace of mind, is that if the Salling Group says they’re going to do something, or if føtex says they’re going to do something or says they want to win this particular battle, then we win it, and then we do it to the full. [...] So when we say, for example, with this health project, that ‘we want to work with health,’ then we do want to work with health, and we’re going to make a difference in health, too. (føtex representative (B))

In this excerpt, she expressed that the mere size of the company allowed them to push any agenda if they wanted to. However, this also underlines that this capacity is dependent on the retailer’s willingness, a willingness that was not in favour of many of the initiatives that the researcher, based on the literature, thought would have the greatest effect.

Even though the size of the company came with many available resources, the retailer also explained how the size of the company had worked against the project in several ways:

What I think made it difficult for us to get through with some of these things let's just take the sodas, in that case, we have a private label collaborator who has production facilities, and when they press the ‘Salling sodas’ button, it doesn't just spew out a few thousand bottles, but millions. So saying ‘can't we just try to reduce the size and give it a try.’ It's a giant setup, so it’s not possible to do that at a whim. You’d need to get a whole or half chain on board that can help sell such volumes because otherwise, the costs would go through the roof. (føtex representative (A))

What this excerpt explains is that even changes that appeared small would take tremendous effort and be very costly, due to the size of the organisation.

Another challenge of the implementation was embedded in the retailers’ organisational culture. Føtex representative (B) explained in the interview that conflicting goals between employees made it difficult and time-consuming when implementing new initiatives. Another barrier to implementing the initiatives was high staff turnover at the retailer. In an interview with a føtex representative, she explained that people often shifted around different positions in the organisation, which ended in the project falling between two stools, leading to misunderstandings of agreements and changes in attitudes towards the initiatives.

In summary, we find that the retailers could, in some respects, have a strong general capacity to implement new initiatives by having available resources and being used to implement new initiatives. Regardless, this study shows that this was not utilised due to a lack of willingness. Moreover, we find that the size and organisational culture of the retailer hampered the implementation of the initiatives.

Influence of customers and other stakeholders on store operation

The last section reports on the influence of customers on the retailer’s willingness to implement the initiatives, and the influence of other stakeholders, especially producers, on what can be implemented.

We found that the customer’s reactions and attitudes were determining to the retailer when implementing any new initiative, as indicated in the sections above. According to the retailer, the customer was the focus when designing the layout of the store:

We are in very close dialogue with our clients, we do quantitative surveys and we do focus groups, we do in-depth interviews. And in that context, we're trying to understand, when you're shopping, how do you go about it. Is it easy for you to find the items you are looking for? And based on the responses, we try to adapt our stores to make things easy for our customers. (føtex representative (A))

The same representative also mentioned that she thought it would have been a strength of the project to have conducted interviews with the customers as a part of the development process, emphasising the weight they put on the customer’s attitudes. The retailers highlighted the importance of customer satisfaction and convenience in their shopping experience as a barrier to implementing certain initiatives, such as changing the placement of products. However, these same factors have also proven to be facilitators for other initiatives, such as the tags for breakfast products and the complimentary banana for children, as demonstrated above.

Another important stakeholder for the supermarkets was the suppliers of their products. Others were government actors (e.g. the Danish Veterinary and Food Administration). For both downsizing initiatives, the suppliers of the products (sodas and bags for sweets) were key to the success of their implementation. In an interview with the store manager, he explained the huge role some of these suppliers have in the daily operation of the store and the chain.

After all, we’ve got a chain agreement that our head office has made with the breweries. I don’t get to decide which items are in our refrigerators. [...] The tricky thing is that we’re not only dealing with føtex or the Salling Group. We also have to do with some other, equally large companies that are also just coming in. Plus, I have people here X times a week to service their particular area. [...] [Another thing] that proved tricky, as far as I recall, was that the alternatives offered, people felt strongly about those because the breweries made some strategic choices, and because of those, some of the items that we might be able to stock, they didn't want to sell those. (Store manager)

This excerpt illustrates how suppliers like the breweries, as shown earlier, influenced the implementation and affected the decisions made by the retailer.

This section indicates that even though the retailer is convinced that a given initiative would be interesting to implement in their supermarket, the suppliers often must agree as well, and finally, the customers must also welcome it.

In this study, we have explored the implementation, acceptability, and feasibility of four different health-promoting food retail initiatives aimed at customers in a real-life supermarket setting, using different qualitative methods. We found that (i) Two initiatives (downsizing of bags for the pick’n’ mix sweets and the complimentary banana for children) were implemented to a high degree, yet delivery issues caused delays according to the planned date, especially for the bags. The downsizing of soda bottles was not implemented as intended; the size and packaging deviated from the original plan due to delivery failure. Moreover, the implementation decreased over the feasibility test for the initiative with shelf tags, as it took more continuous maintenance. For all initiatives, we found that they lingered after the feasibility test; however, only the banana for children was somewhat sustained for a period to accommodate customer demand. (ii) The retailers expressed different levels of acceptability towards the initiatives, and different representatives sometimes also showed different levels of acceptability towards the same initiative, such as the tags on the breakfast products. The most well-received initiative was the banana for children, which is somewhat unsurprising, as it was the retailers themselves that suggested including this initiative. Additionally, the positive response from the customers that they got supported the retailers’ positive attitude towards the initiative. We also found that many customers responded well to this initiative; however, we also observed a group that did not accept the initiative and preferred the bun over the banana. For the remaining initiatives, customers did not seem to notice them. Yet, we did observe customer behaviours that would probably work against the suggested mechanisms of some of the initiatives. (iii) In general, we describe three themes of barriers and facilitators that influence the implementation and possible sustainment of the initiatives: Health is not the number one priority, General capacity of the retailer, and Influence of customers and other stakeholders on store operation. Firstly, we found the retailers were motivated to work with health, both from a personal and professional perspective. The motivation was rooted in a feeling of social responsibility as well as health initiatives being viewed as a relative advantage, due to demand and making customers happier. Still, other priorities, such as profit and maintaining customers’ ‘free choice’, challenged the motivation to implement such initiatives. Secondly, the retailer showed a high level of available resources, which supported their general capacity to implement the initiatives; however, the large size of the organisation and its culture proved to be barriers to the implementation. Lastly, the analysis showed that the influence of both customers and other stakeholders was crucial to the implementation, both in terms of what is possible and what the retailers would be interested in and prioritise.

Our findings are similar to those of others [ 26 , 35 ]. Winkler et al. [ 35 ] found that even though supermarket actors found health-promoting initiatives meaningful to engage in, their engagement was challenged by a business mindset, practical routines, and structural requirements. Thus, despite the involvement of retailers in the development, selection and implementation of the initiatives, studies suggest that healthy food retail initiatives still encounter some fundamental barriers towards the implementation, such as the economical aspect or the view on customers’ free choice. However, our results also indicate that it might be possible to persuade food retailers to remove products or restrict choices if the evidence or arguments of it being the right thing to do are sufficiently strong, as with organic bananas or tobacco products. This has also been the case of another retailer in Denmark, which has decided that all their stores should be tobacco and nicotine-free by the end of 2028 to reduce the number of smokers [ 36 ]. Another solution is to identify win–win initiatives, as the complimentary banana for children was somewhat an example of (if we consider the banana as a healthier alternative) and which other studies have found as well [ 35 , 37 ].

Even though the four initiatives were implemented (yet two not as intended) in this study, and we found them to be somewhat acceptable to the retailers, we must still highlight that these initiatives represent a very small portion of the initiatives first suggested and entail several compromises from what the researchers had initially planned (Duus et al. Unpublished ). Moreover, the customer’s responses to the initiatives were mixed, and in some cases, their behaviour indicated that the initiatives would have little effect. Compared with studies testing similar initiatives, we find that 1) Shelf tags alone were found unlikely to change food purchases [ 38 ] and are likely to contribute to disparities in food purchases as not all customers know nutrition labels or have the literacy to read and understand them [ 39 ]. 2) Smaller bags for pick’n’ mix sweets could be successfully implemented and, based on results from another study, might be able to decrease the volume of sweets sold [ 40 ]. Moreover, others have also shown that customers are willing to buy smaller product options [ 41 ]. Taken together, this suggests that voluntary engagement with researchers might not suffice to make changes that would improve the supermarket environment as opted for to support population health. This view has also been suggested by Winkler et al. [ 35 ], and in the Lancet series on commercial determinants of health, an even more critical perspective on engagements with commercial actors as food retailers is presented [ 42 , 43 ]. Here they warn against how commercial actors use partnerships with researchers, among others, as a tool to improve their reputation and credibility [ 42 ].

In our collaborative process with the retailer, we experienced many challenges. We did not accomplish aligning retailers’ and researchers’ interests as scholars have suggested being the prerequisite of implementing healthy food retail interventions in supermarkets [ 26 , 27 ]. This underlines the importance of the pre-intervention phase, as described by Hawe, Shiell, and Riley [ 44 ], which is fundamental to a successful implementation. During the pre-intervention phase, the establishment of relationships between different people or agencies often occurs, and these relationships may play a crucial role in the implementation and the explanation of why some work and others do not [ 44 ]. In line with this, another study has suggested exploring what implementation strategies might promote the uptake of evidence-based interventions among food retailers [ 45 ]. They found that contrary to many other studies, the intervention in their study was compatible with the interest of the store managers to which it was presented—these store managers had a strong feeling of social responsibility towards the communities they operated in [ 45 ].

Strength and limitations

The investigation of the feasibility test was strengthened by using different methods, process evaluation concepts, and a broad view including both the delivery and presentation of the initiatives as well as customer and retailer perspectives. We primarily got the retailer perspective from a strategic level, yet we had planned on conducting focus group interviews with staff at the test store to get perspectives from an operational level on the initiatives and the implementation process. However, no staff wanted to participate in an interview. The store manager explained that this probably was due to three things: 1) They had no interest in the study, or they were tired of the study, 2) the recruitment was done too late (approximately 2 months after the feasibility test period), and 3) the staff was overworked as a result of understaffing due to the coronavirus disease pandemic. Future studies aim also to analyse sales data in order to evaluate whether any changes in sales of the products we intervened on occurred. However, with the available data, we will not be able to analyse whether the initiatives change people’s eating patterns or whether they influence people differently in terms of their socioeconomic factors or other characteristics.

A thorough needs assessment [ 46 ] among supermarket customers to test the initiative’s assumptions and their food purchase patterns would have strengthened the study. However, this was not possible within the timeframe and funding scheme, so the development drew primarily on existing knowledge and the experience of the retailer and the Danish Cancer Society. Furthermore, the store visits conducted in the store during the development of the initiative also provided a few customer perspectives, which led to the exclusion of some ideas (Duus et al.  unpublished ).

Furthermore, we learned two methodological lessons from the in-store observations: 1) All observers were met by the feeling of being ‘in the way’ and a need to be in almost constant movement to not interfere with the order in the store. The observers were met with a feeling of self-awareness and a need to legitimise their presence at the store by wearing a sticker on their shirts saying ‘visitor’ or their university identification card. These feelings were amplified by the governmental advice of social distancing and the requirement to wear face masks in grocery stores, introduced during the period of observations. 2) Concerning this, the observers also found it challenging to approach customers for the short interviews due to the feeling of invading people’s private space, hence only five were conducted. This was especially challenging when wearing face masks, as it was impossible to produce and read non-verbal signals (e.g. smiles), and difficult to hear what people were saying.

Implications for future studies and practice

This study presents an investigation of the implementation of healthy food retail initiatives for supermarkets that have been developed and selected together with retailers as suggested by the literature. It suggests that the implementation of such initiatives is possible and—to some degree—high. Yet, the quality of the initiatives was rather low, and some were not implemented as intended. Moreover, we still present some of the same barriers and limitations as former studies that have not implemented collaborative strategies in the pre-intervention phase. Some of this may be due to challenges such as a high staff turnover at the retailer and a lack of a shared understanding, as shown in another study (Duus et al. unpublished ). Future studies must explore this further.

Lessons for future studies are to identify initiatives that customers appreciate, as this is important to retailers. Underlining a needs assessment as an important first step in intervention development [ 30 , 46 ]. Furthermore, future studies should involve a broader range of stakeholders, including manufacturers and suppliers, in the development of the initiatives, as they have significant power over what can be implemented. Future studies would also benefit from identifying and testing implementation strategies that can facilitate the implementation of this type of intervention in this setting.

We performed a qualitative investigation of the implementation, acceptability, and feasibility of four different healthy food retail initiatives aimed at customers in a real-life supermarket setting, which had been developed and selected together with retailers. Only two of the four initiatives were implemented as intended, and the perspectives of retailers and customers were mixed or unclear. Altogether, the study highlights the challenges of implementing healthy retail food initiatives despite early involvement of retailers in the selection and design of those initiatives. Adding to the challenges of implementation, the initiatives also represent a compromise between the interests of the researcher and the retailers and do not represent what the literature suggests as the most effective strategies. A compromise made to uphold the partnership and complete the funded research project. Future studies should further examine the impact and pitfalls of including retailers (or other commercial actors) in the development and selection of healthy food retail initiatives and try to identify successful implementation strategies facilitating implementation.

Availability of data and materials

The data generated and analysed during the current study are not publicly available due to their sensitive and confidential nature but are available from the corresponding author upon reasonable request.

Abbreviations

Corporate Social Responsibility

Key Performance Indicator

Healthy diet. WHO. 2020. https://www.who.int/news-room/fact-sheets/detail/healthy-diet . Accessed 20 July 2023.

Greenwald P, Clifford CK, Milner JA. Diet and cancer prevention. Eur J Cancer. 2001May 1;37(8):948–65.

Article   PubMed   CAS   Google Scholar  

Firth J, Gangwisch JE, Borsini A, Wootton RE, Mayer EA. Food and mood: how do diet and nutrition affect mental wellbeing? BMJ. 2020Jun;29(369): m2382.

Article   Google Scholar  

English LK, Ard JD, Bailey RL, Bates M, Bazzano LA, Boushey CJ, et al. Evaluation of Dietary Patterns and All-Cause Mortality. JAMA Netw Open. 2021Aug 31;4(8): e2122277.

Article   PubMed   PubMed Central   Google Scholar  

Swinburn B, Caterson I, Seidell J, James W. Diet, nutrition and the prevention of excess weight gain and obesity. Public Health Nutr. 2004Feb;7(1a):123–46.

Brug J. Determinants of healthy eating: motivation, abilities and environmental opportunities. Fam Pract. 2008 Dec 1;25(suppl_1):i50–5.

Adam A, Jensen JD. What is the effectiveness of obesity related interventions at retail grocery stores and supermarkets? —a systematic review. BMC Public Health. 2016Dec;16(1):1247.

Ball K, Timperio AF, Crawford DA. Understanding environmental influences on nutrition and physical activity behaviors: where should we look and what should we count? Int J Behav Nutr Phys Act. 2006Sep 26;3(1):33.

Sonntag D, Schneider S, Mdege N, Ali S, Schmidt B. Beyond Food Promotion: A Systematic Review on the Influence of the Food Industry on Obesity-Related Dietary Behaviour among Children. Nutrients. 2015;7(10):8565–76.

Viola D, Arno PS, Maroko AR, Schechter CB, Sohler N, Rundle A, et al. Overweight and obesity: Can we reconcile evidence about supermarkets and fast food retailers for public health policy? J Public Health Policy. 2013Aug;34(3):424–38.

Black C, Moon G, Baird J. Dietary inequalities: What is the evidence for the effect of the neighbourhood food environment? Health Place. 2014May;27:229–42.

Article   PubMed   Google Scholar  

Vogel C, Ntani G, Inskip H, Barker M, Cummins S, Cooper C, et al. Education and the Relationship Between Supermarket Environment and Diet. Am J Prev Med. 2016Aug;51(2):e27-34.

Chandon P, Wansink B. Does food marketing need to make us fat? A review and solutions Nutr Rev. 2012;70(10):571–93.

PubMed   Google Scholar  

Bennett R, Zorbas C, Huse O, Peeters A, Cameron AJ, Sacks G, et al. Prevalence of healthy and unhealthy food and beverage price promotions and their potential influence on shopper purchasing behaviour: A systematic review of the literature. Obes Rev. 2020 Jan;21(1).

Harbers MC, Beulens JWJ, Rutters F, de Boer F, Gillebaart M, Sluijs I, et al. The effects of nudges on purchases, food choice, and energy intake or content of purchases in real-life food purchasing environments: a systematic review and evidence synthesis. Nutr J. 2020Dec;19(1):103.

Hollands GJ, Carter P, Anwer S, King SE, Jebb SA, Ogilvie D, et al. Altering the availability or proximity of food, alcohol, and tobacco products to change their selection and consumption. Cochrane Public Health Group, editor. Cochrane Database Syst Rev. 2019 Sep 4.

Slapø H, Schjøll A, Strømgren B, Sandaker I, Lekhal S. Efficiency of In-Store Interventions to Impact Customers to Purchase Healthier Food and Beverage Products in Real-Life Grocery Stores: A Systematic Review and Meta-Analysis. Foods. 2021May;10(5):922.

Gravlee CC, Boston PQ, Mitchell MM, Schultz AF, Betterley C. Food store owners’ and managers’ perspectives on the food environment: an exploratory mixed-methods study. BMC Public Health. 2014Dec;14(1):1031.

Winkler LL, Christensen U, Glümer C, Bloch P, Mikkelsen BE, Wansink B, et al. Substituting sugar confectionery with fruit and healthy snacks at checkout – a win-win strategy for consumers and food stores? a study on consumer attitudes and sales effects of a healthy supermarket intervention. BMC Public Health. 2016Nov 22;16(1):1184.

Adam A, Jensen JD, Sommer I, Hansen GL. Does shelf space management intervention have an effect on calorie turnover at supermarkets? J Retail Consum Serv. 2017Jan;1(34):311–8.

Toft U, Buch-Andersen T, Bloch P, Reinbach HC, Jensen BB, Mikkelsen BE, et al. A Community-Based, Participatory, Multi-Component Intervention Increased Sales of Healthy Foods in Local Supermarkets—The Health and Local Community Project (SoL). Int J Environ Res Public Health. 2023Jan;20(3):2478.

Bauer JM, Aarestrup SC, Hansen PG, Reisch LA. Nudging more sustainable grocery purchases: Behavioural innovations in a supermarket setting. Technol Forecast Soc Change. 2022Jun;1(179): 121605.

Denver S, Christensen T, Nordström J. Consumer preferences for low-salt foods: a Danish case study based on a comprehensive supermarket intervention. Public Health Nutr. 2021;24(12):3956–65.

Jensen JD, Sommer I. Reducing calorie sales from supermarkets – ‘silent’ reformulation of retailer-brand food products. Int J Behav Nutr Phys Act. 2017Aug 23;14(1):104.

Toft U, Winkler LL, Mikkelsen BE, Bloch P, Glümer C. Discounts on fruit and vegetables combined with a space management intervention increased sales in supermarkets. Eur J Clin Nutr. 2017Apr;71(4):476–80.

Middel CNH, Schuitmaker-Warnaar TJ, Mackenbach JD, Broerse JEW. Systematic review: a systems innovation perspective on barriers and facilitators for the implementation of healthy food-store interventions. Int J Behav Nutr Phys Act. 2019Dec;16(1):108.

Gupta A, Alston L, Needham C, Robinson E, Marshall J, Boelsen-Robinson T, et al. Factors Influencing Implementation, Sustainability and Scalability of Healthy Food Retail Interventions: A Systematic Review of Reviews. Nutrients. 2022Jan;14(2):294.

Denmark: market share of grocery retailers 2020. https://www.statista.com/statistics/565747/market-share-of-selected-grocery-retailers-in-denmark/ . Accessed 19 July 2023.

Moore N, editor. Desk research. In: How to Do Research: The Practical Guide to Designing and Managing Research Projects. Facet; 2006. p. 106–11.

Schultz Petersen K, Maindal HT, Ledderer L, Overgaard C. Komplekse interventioner: Udvikling, test, evaluering og implementering. Aalborg Universitetsforlag; 2022.

McGill E, Marks D, Er V, Penney T, Petticrew M, Egan M. Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators. PLoS Med. 2020Nov 2;17(11): e1003368.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006Jan;3(2):77–101.

Scaccia JP, Cook BS, Lamont A, Wandersman A, Castellow J, Katz J, et al. A practical implementation science heuristic for organizational readiness: R = MC2. J Community Psychol. 2015Apr;43(4):484–501.

Blake MR, Backholer K, Lancsar E, Boelsen-Robinson T, Mah C, Brimblecombe J, et al. Investigating business outcomes of healthy food retail strategies: A systematic scoping review. Obes Rev. 2019Oct;20(10):1384–99.

Winkler LL, Toft U, Glümer C, Bloch P, Buch-Andersen T, Christensen U. Involving supermarkets in health promotion interventions in the Danish Project SoL. A practice-oriented qualitative study on the engagement of supermarket staff and managers. BMC Public Health. 2023 Apr 18;23(1):706.

Lidl Danmark. https://om.lidl.dk/ansvarlighed/vi-fremmer-sundheden/udfasning-af-tobak . Accessed 7 March 2024.

Blake MR, Sacks G, Zorbas C, Marshall J, Orellana L, Brown AK, et al. The ‘Eat Well @ IGA’ healthy supermarket randomised controlled trial: process evaluation. Int J Behav Nutr Phys Act. 2021Dec;18(1):36.

Vandevijvere S, Berger N. The impact of shelf tags with Nutri-Score on consumer purchases: a difference-in-difference analysis of a natural experiment in supermarkets of a major retailer in Belgium. Int J Behav Nutr Phys Act. 2021Nov 18;18(1):150.

Robertson A, Lobstein T, Knai C. Obesity and socio-economic groups in Europe: Evidence review and implications for action. 2007.

Mørck CJ. Nyt forsøg afslører: Bland selv-posens størrelse gør en stor forskel. 2024. https://www.cancer.dk/nyheder-og-fortaellinger/2024/nyt-forsoeg-afsloerer-bland-selv-posens-stoerrelse-goer-en-stor-forskel/ . Accessed 4 July 2024.

Vandenbroele J, Slabbinck H, Van Kerckhove A, Vermeir I. Curbing portion size effects by adding smaller portions at the point of purchase. Food Qual Prefer. 2018Mar;1(64):82–7.

Gilmore AB, Fabbri A, Baum F, Bertscher A, Bondy K, Chang HJ, et al. Defining and conceptualising the commercial determinants of health. Lancet. 2023Apr;401(10383):1194–213.

Lacy-Nichols J, Nandi S, Mialon M, McCambridge J, Lee K, Jones A, et al. Conceptualising commercial entities in public health: beyond unhealthy commodities and transnational corporations. Lancet. 2023Apr;401(10383):1214–28.

Hawe P, Shiell A, Riley T. Theorising Interventions as Events in Systems. Am J Community Psychol. 2009;43(3–4):267–76.

Brimblecombe J, Miles B, Chappell E, De Silva K, Ferguson M, Mah C, et al. Implementation of a food retail intervention to reduce purchase of unhealthy food and beverages in remote Australia: mixed-method evaluation using the consolidated framework for implementation research. Int J Behav Nutr Phys Act. 2023Feb 17;20(1):20.

Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021Sep;30(374): n2061.

Download references

Acknowledgements

We want to thank all the participating retail group and supermarket staff members involved in this project and the implementation process. We appreciate the time and effort you have dedicated to this project and your openness. Furthermore, we want to acknowledge the customers who took the time to share their opinions with us during their daily grocery shopping.

We acknowledge Johanne Aviaja Rosing, Louise Ayoe Sparvath Brautsch, and Carl Johannes Middelboe for their assistance in conducting the pre- and post-intervention observations.

Open access funding provided by University of Southern Denmark This study is funded by the Danish Cancer Society, grant no.: R274-A16920. The first author (Katrine Sidenius Duus) has also received a Faculty Scholarship from the Faculty of Health Sciences at the University of Southern Denmark to support the completion of her PhD thesis, which this study is part of.

Author information

Authors and affiliations.

The National Institute of Public Health, University of Southern Denmark, Copenhagen, Denmark

Katrine Sidenius Duus, Tine Tjørnhøj-Thomsen & Rikke Fredenslund Krølner

You can also search for this author in PubMed   Google Scholar

Contributions

KSD, RFK, and TTT contributed to the funding acquisition, study conception and design. Data generation and analyses were performed by KSD. The first draft of the manuscript was written by KSD. RFK and TTT commented on previous versions of the manuscript and contributed in writing the final manuscript. KSD wrote up the final manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Katrine Sidenius Duus .

Ethics declarations

Ethics approval and consent to participate.

This study has been approved by SDU Research & Innovation Organization (notification no. 11.136). All informants who participated in interviews received written and verbal information about the aim, that participation was voluntary and that their information would be used for research purposes only and treated with confidentiality. By participating, consent for their data to be used for research was given. Data from the observation and documents were handled confidentially and with caution to protect sensitive information that could identify individuals.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Duus, K.S., Tjørnhøj-Thomsen, T. & Krølner, R.F. Implementation of health-promoting retail initiatives in the Healthier Choices in Supermarkets Study—qualitative perspectives from a feasibility study. BMC Med 22 , 349 (2024). https://doi.org/10.1186/s12916-024-03561-2

Download citation

Received : 24 May 2024

Accepted : 14 August 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s12916-024-03561-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation
• Qualitative research
• Health promotion
• Supermarkets
• Involvement
• Intervention

BMC Medicine

ISSN: 1741-7015

• General enquiries: [email protected]