mixed methods research literature review

USF Libraries Hours by campus

Search the usf libraries, libraries locations.

• Libraries Hours
• Outages & Maintenance Alerts
• Grad Students
• Alumni & Public
• Ask A Librarian
• Subject & Course Guides
• Research Tools
• Find My Librarian

COLLECTIONS

• Special Collections
• Digital Initiatives
• Digital Commons @ USF
• Center for Digital Heritage & Geospatial Information
• Administration
• Get Help With...
• USF Health Libraries

Systematic Reviews for Social Sciences

• Systematic Reviews
• Mapping Review

Mixed Methods Review

• Rapid Review
• Scoping Review
• Develop the Question
• Define Inclusion/Exclusion Criteria
• Develop A Review Protocol
• Create Search Strategies
• Select Studies
• Extract Data
• Assess the Quality of Studies
• Synthesize Data and Write the Report
• Review Tools
• Librarian Involvement
• Grey Literature

A  comprehensive syntheses of two or more types of data (e.g. quantitative and qualitative) are conducted and then aggregated into a final, combined synthesis

Mixed Methods Reviews are best designed for:

• Multidisciplinary topics or topics with a body of literature that includes quantitative, qualitative, and mixed methods studies.
• To determine not only the effects of interventions but also their appropriateness.
• To identify research gaps.
• To provide an explanation for possible heterogeneity between trials.
• To answer multiple questions in one systematic review.

Limitations:

• May result in a larger number of citations.
• Requires more search time to create multiple searches for varying outcomes.
• There are no universally adopted methods for conducting Mixed Methods Reviews.
• Requires significant methodological skill.
• Resource intensive--may take time to engage with the evidence and develop theory.
• Not inherently reproducible or transparent because of the highly iterative nature of the interpretative process

(Source: M. Petticrew et al, 2013)

Outline of Stages

Timeframe: 12-18+ months.  Same as a systematic review or longer.  

Question: Addresses 2 or more review framework criteria, "Mixed methods reviews should pose a question that specifically requires the inclusion of two or more syntheses that are grounded in different approaches".  A priori review protocol is recommended.

Examples of and clearly articulated questions that may be posed by one mixed methods systematic review are:

1. What is the effectiveness of educational strategies associated with insulin pump therapy?

2. What is the appropriateness of educational strategies associated with insulin pump therapy

Sources and searches: Types of studies and their findings are mixed.  This requires a very broad search or multiple structured searches. "This should address each of the syntheses included in the review...[and] aims to find both published and unpublished studies."

Selection: Based upon inclusion criteria.  

• Quantitative component -  consider any experimental study design before and after studies for inclusion
• Qualitative component - consider studies that focus on qualitative data including, but not limited to, designs such as phenomenology, grounded theory, ethnography, action research and feminist research. In the absence of research studies, other text such as opinion papers and reports should be considered.
• Textual component - consider expert opinion, discussion papers, position papers and other text
•  Economic component - consider cost effectiveness, cost benefit, cost minimization, cost utility

Appraisal:   Address each of the syntheses included in the review

Example: Quantitative, Qualitative, Textual or Economic papers

The appropriate critical appraisal tool or method should be applied by study type or there should be one general tool used that will represent all study designs.

Synthesis: Depends on findings and included studies, applying mixed synthesis methods (qualitative: thematic synthesis/meta-ethnography & quantitative: statistical meta-analysis).  “The results of each single method synthesis included in the mixed method review will be extracted in numerical, tabular or textual format."

(Source: Joanna Briggs Institute 2014)

• JBI Manual: Mixed Methods Review
• Toolkit for Mixed Studies Reviews
• Using mixed methods research synthesis for literature reviews
• Conducting Mixed Methods Research Systematic Methodological Reviews: A Review of Practice and Recommendations
• Text-in-context: a method for extracting findings in mixed-methods mixed research synthesis studies
• Mixed-Methods Systematic Reviews: Integrating Quantitative and Qualitative Findings
• Mixed Methods Appraisal Tool (MMAT)
• Mixed Methods Studies in Environmental Management Research: Prevalence, Purposes and Designs
• Help seeking by male victims of domestic violence and abuse: an example of an integrated mixed methods synthesis of systematic review evidence defining methodological terms
• Community-Led Total Sanitation: A Mixed-Methods Systematic Review of Evidence and Its Quality
• << Previous: Mapping Review
• Next: Rapid Review >>
• Last Updated: Jun 11, 2024 8:21 PM
• URL: https://guides.lib.usf.edu/systematicreviews

Mixed methods research: what it is and what it could be

• Open access
• Published: 29 March 2019
• Volume 48 , pages 193–216, ( 2019 )

Cite this article

You have full access to this open access article

• Rob Timans 1 ,
• Paul Wouters 2 &
• Johan Heilbron 3  

129k Accesses

111 Citations

12 Altmetric

Explore all metrics

A Correction to this article was published on 06 May 2019

This article has been updated

Combining methods in social scientific research has recently gained momentum through a research strand called Mixed Methods Research (MMR). This approach, which explicitly aims to offer a framework for combining methods, has rapidly spread through the social and behavioural sciences, and this article offers an analysis of the approach from a field theoretical perspective. After a brief outline of the MMR program, we ask how its recent rise can be understood. We then delve deeper into some of the specific elements that constitute the MMR approach, and we engage critically with the assumptions that underlay this particular conception of using multiple methods. We conclude by offering an alternative view regarding methods and method use.

Similar content being viewed by others

Mixed Methods

Explore related subjects

• Artificial Intelligence

Avoid common mistakes on your manuscript.

The interest in combining methods in social scientific research has a long history. Terms such as “triangulation,” “combining methods,” and “multiple methods” have been around for quite a while to designate using different methods of data analysis in empirical studies. However, this practice has gained new momentum through a research strand that has recently emerged and that explicitly aims to offer a framework for combining methods. This approach, which goes by the name of Mixed Methods Research (MMR), has rapidly become popular in the social and behavioural sciences. This can be seen, for instance, in Fig.  1 , where the number of publications mentioning “mixed methods” in the title or abstract in the Thomson Reuters Web of Science is depicted. The number increased rapidly over the past ten years, especially after 2006. Footnote 1

Fraction of the total of articles mentioning Mixed Method Research appearing in a given year, 1990–2017 (yearly values sum to 1). See footnote 1

The subject of mixed methods thus seems to have gained recognition among social scientists. The rapid rise of the number of articles mentioning the term raises various sociological questions. In this article, we address three of these questions. The first question concerns the degree to which the approach of MMR has become institutionalized within the field of the social sciences. Has MMR become a recognizable realm of knowledge production? Has its ascendance been accompanied by the production of textbooks, the founding of journals, and other indicators of institutionalization? The answer to this question provides an assessment of the current state of MMR. Once that is determined, the second question is how MMR’s rise can be understood. Where does the approach come from and how can its emergence and spread be understood? To answer this question, we use Pierre Bourdieu’s field analytical approach to science and academic institutions (Bourdieu 1975 , 1988 , 2004 , 2007 ; Bourdieu et al. 1991 ). We flesh out this approach in the next section. The third question concerns the substance of the MMR corpus seen in the light of the answers to the previous questions: how can we interpret the specific content of this approach in the context of its socio-historical genesis and institutionalization, and how can we understand its proposal for “mixing methods” in practice?

We proceed as follows. In the next section, we give an account of our theoretical approach. Then, in the third, we assess the degree of institutionalization of MMR, drawing on the indicators of academic institutionalization developed by Fleck et al. ( 2016 ). In the fourth section, we address the second question by examining the position of the academic entrepreneurs behind the rise of MMR. The aim is to understand these agents’ engagement in MMR, as well as its distinctive content as being informed by their position in this field. Viewing MMR as a position-taking of academic entrepreneurs, linked to their objective position in this field, allows us to reflect sociologically on the substance of the approach. We offer this reflection in the fifth section, where we indicate some problems with MMR. To get ahead of the discussion, these problems have to do with the framing of MMR as a distinct methodology and its specific conceptualization of data and methods of data analysis. We argue that these problems hinder fruitfully combining methods in a practical understanding of social scientific research. Finally, we conclude with some tentative proposals for an alternative view on combining methods.

A field approach

Our investigation of the rise and institutionalization of MMR relies on Bourdieu’s field approach. In general, field theory provides a model for the structural dimensions of practices. In fields, agents occupy a position relative to each other based on the differences in the volume and structure of their capital holdings. Capital can be seen as a resource that agents employ to exert power in the field. The distribution of the form of capital that is specific to the field serves as a principle of hierarchization in the field, differentiating those that hold more capital from those that hold less. This principle allows us to make a distinction between, respectively, the dominant and dominated factions in a field. However, in mature fields all agents—dominant and dominated—share an understanding of what is at stake in the field and tend to accept its principle of hierarchization. They are invested in the game, have an interest in it, and share the field’s illusio .

In the present case, we can interpret the various disciplines in the social sciences as more or less autonomous spaces that revolve around the shared stake in producing legitimate scientific knowledge by the standards of the field. What constitutes legitimate knowledge in these disciplinary fields, the production of which bestows scholars with prestige and an aura of competence, is in large part determined by the dominant agents in the field, who occupy positions in which most of the consecration of scientific work takes place. Scholars operating in a field are endowed with initial and accumulated field-specific capital, and are engaged in the struggle to gain additional capital (mainly scientific and intellectual prestige) in order to advance their position in the field. The main focus of these agents will generally be the disciplinary field in which they built their careers and invested their capital. These various disciplinary spaces are in turn part of a broader field of the social sciences in which the social status and prestige of the various disciplines is at stake. The ensuing disciplinary hierarchy is an important factor to take into account when analysing the circulation of new scientific products such as MMR. Furthermore, a distinction needs to be made between the academic and the scientific field. While the academic field revolves around universities and other degree-granting institutions, the stakes in the scientific field entail the production and valuation of knowledge. Of course, in modern science these fields are closely related, but they do not coincide (Gingras and Gemme 2006 ). For instance, part of the production of legitimate knowledge takes place outside of universities.

This framework makes it possible to contextualize the emergence of MMR in a socio-historical way. It also enables an assessment of some of the characteristics of MMR as a scientific product, since Bourdieu insists on the homology between the objective positions in a field and the position-takings of the agents who occupy these positions. As a new methodological approach, MMR is the result of the position-takings of its producers. The position-takings of the entrepreneurs at the core of MMR can therefore be seen as expressions in the struggles over the authority to define the proper methodology that underlies good scientific work regarding combining methods, and the potential rewards that come with being seen, by other agents, as authoritative on these matters. Possible rewards include a strengthened autonomy of the subfield of MMR and an improved position in the social-scientific field.

The role of these entrepreneurs or ‘intellectual leaders’ who can channel intellectual energy and can take the lead in institution building has been emphasised by sociologists of science as an important aspect of the production of knowledge that is visible and recognized as distinct in the larger scientific field (e.g., Mullins 1973 ; Collins 1998 ). According to Bourdieu, their position can, to a certain degree, explain the strategy they pursue and the options they perceive to be viable in the trade-off regarding the risks and potential rewards for their work.

We do not provide a full-fledged field analysis of MMR here. Rather, we use the concept as a heuristic device to account for the phenomenon of MMR in the social context in which it emerged and diffused. But first, we take stock of the current situation of MMR by focusing on the degree of institutionalization of MMR in the scientific field.

The institutionalization of mixed methods research

When discussing institutionalization, we have to be careful about what we mean by this term. More precisely, we need to be specific about the context and distinguish between institutionalization in the academic field and institutionalization within the scientific field (see Gingras and Gemme 2006 ; Sapiro et al. 2018 ). The first process refers to the establishment of degrees, curricula, faculties, etc., or to institutions tied to the academic bureaucracy and academic politics. The latter refers to the emergence of institutions that support the autonomization of scholarship such as scholarly associations and scientific journals. Since MMR is still a relatively young phenomenon and academic institutionalization tends to lag scientific institutionalization (e.g., for the case of sociology and psychology, see Sapiro et al. 2018 , p. 26), we mainly focus here on the latter dimension.

Drawing on criteria proposed by Fleck et al. ( 2016 ) for the institutionalization of academic disciplines, MMR seems to have achieved a significant degree of institutionalization within the scientific field. MMR quickly gained popularity in the first decade of the twenty-first century (e.g., Tashakkori and Teddlie 2010c , pp. 803–804). A distinct corpus of publications has been produced that aims to educate those interested in MMR and to function as a source of reference for researchers: there are a number of textbooks (e.g., Plowright 2010 ; Creswell and Plano Clark 2011 ; Teddlie and Tashakkori 2008 ); a handbook that is now in its second edition (Tashakkori and Teddlie 2003 , 2010a ); as well as a reader (Plano Clark and Creswell 2007 ). Furthermore, a journal (the Journal of Mixed Methods Research [ JMMR] ) was established in 2007. The JMMR was founded by the editors John Creswell and Abbas Tashakkori with the primary aim of “building an international and multidisciplinary community of mixed methods researchers.” Footnote 2 Contributions to the journal must “fit the definition of mixed methods research” Footnote 3 and explicitly integrate qualitative and quantitative aspects of research, either in an empirical study or in a more theoretical-methodologically oriented piece.

In addition, general textbooks on social research methods and methodology now increasingly devote sections to the issue of combining methods (e.g., Creswell 2008 ; Nagy Hesse-Biber and Leavy 2008 ; Bryman 2012 ), and MMR has been described as a “third paradigm” (Denscombe 2008 ), a “movement” (Bryman 2009 ), a “third methodology” (Tashakkori and Teddlie 2010b ), a “distinct approach” (Greene 2008 ) and an “emerging field” (Tashakkori and Teddlie 2011 ), defined by a common name (that sets it apart from other approaches to combining methods) and shared terminology (Tashakkori and Teddlie 2010b , p. 19). As a further indication of institutionalization, a research association (the Mixed Methods International Research Association—MMIRA) was founded in 2013 and its inaugural conference was held in 2014. Prior to this, there have been a number of conferences on MMR or occasions on which MMR was presented and discussed in other contexts. An example of the first is the conference on mixed method research design held in Basel in 2005. Starting also in 2005, the British Homerton School of Health Studies has organised a series of international conferences on mixed methods. Moreover, MMR was on the list of sessions in a number of conferences on qualitative research (see, e.g., Creswell 2012 ).

Another sign of institutionalization can be found in efforts to forge a common disciplinary identity by providing a narrative about its history. This involves the identification of precursors and pioneers as well as an interpretation of the process that gave rise to a distinctive set of ideas and practices. An explicit attempt to chart the early history of MMR is provided by Johnson and Gray ( 2010 ). They frame MMR as rooted in the philosophy of science, particularly as a way of thinking about science that has transcended some of the most salient historical oppositions in philosophy. Philosophers like Aristotle and Kant are portrayed as thinkers who sought to integrate opposing stances, forwarding “proto-mixed methods ideas” that exhibited the spirit of MMR (Johnson and Gray 2010 , p. 72, p. 86). In this capacity, they (as well as other philosophers like Vico and Montesquieu) are presented as part of MMR providing a philosophical validation of the project by presenting it as a continuation of ideas that have already been voiced by great thinkers in the past.

In the second edition of their textbook, Creswell and Plano Clark ( 2011 ) provide an overview of the history of MMR by identifying five historical stages: the first one being a precursor to the MMR approach, consisting of rather atomised attempts by different authors to combine methods in their research. For Creswell and Plano Clark, one of the earliest examples is Campbell and Fiske’s ( 1959 ) combination of quantitative methods to improve the validity of psychological scales that gave rise to the triangulation approach to research. However, they regard this and other studies that combined methods around that time, as “antecedents to (…) more systematic attempts to forge mixed methods into a complete research design” (Creswell and Plano Clark 2011 , p. 21), and hence label this stage as the “formative period” (ibid., p. 25). Their second stage consists of the emergence of MMR as an identifiable research strand, accompanied by a “paradigm debate” about the possibility of combining qualitative and quantitative data. They locate its beginnings in the late 1980s when researchers in various fields began to combine qualitative and quantitative methods (ibid., pp. 20–21). This provoked a discussion about the feasibility of combining data that were viewed as coming from very different philosophical points of view. The third stage, the “procedural development period,” saw an emphasis on developing more hands-on procedures for designing a mixed methods study, while stage four is identified as consisting of “advocacy and expansion” of MMR as a separate methodology, involving conferences, the establishment of a journal and the first edition of the aforementioned handbook (Tashakkori and Teddlie 2003 ). Finally, the fifth stage is seen as a “reflective period,” in which discussions about the unique philosophical underpinnings and the scientific position of MMR emerge.

Creswell and Plano Clark thus locate the emergence of “MMR proper” at the second stage, when researchers started to use both qualitative and quantitative methods within a single research effort. As reasons for the emergence of MMR at this stage they identify the growing complexity of research problems, the perception of qualitative research as a legitimate form of inquiry (also by quantitative researchers) and the increasing need qualitative researchers felt for generalising their findings. They therefore perceive the emergence of the practice of combining methods as a bottom up process that grew out of research practices, and at some point in time converged towards a more structural approach. Footnote 4 Historical accounts such as these add a cognitive dimension to the efforts to institutionalize MMR. They lay the groundwork for MMR as a separate subfield with its own identity, topics, problems and intellectual history. The use of terms such as “third paradigm” and “third methodology” also suggests that there is a tendency to perceive and promote MMR as a distinct and coherent way to do research.

In view of the brief exploration of the indicators of institutionalisation of MMR, it seems reasonable to conclude that MMR has become a recognizable and fairly institutionalized strand of research with its own identity and profile within the social scientific field. This can be seen both from the establishment of formal institutions (like associations and journals) and more informal ones that rely more on the tacit agreement between agents about “what MMR is” (an example of this, which we address later in the article, is the search for a common definition of MMR in order to fix the meaning of the term). The establishment of these institutions supports the autonomization of MMR and its emancipation from the field in which it originated, but in which it continues to be embedded. This way, it can be viewed as a semi-autonomous subfield within the larger field of the social sciences and as the result of a differentiation internal to this field (Steinmetz 2016 , p. 109). It is a space that is clearly embedded within this higher level field; for example, members of the subfield of MMR also qualify as members of the overarching field, and the allocation of the most valuable and current form of capital is determined there as well. Nevertheless, as a distinct subfield, it also has specific principles that govern the production of knowledge and the rewards of domination.

We return to the content and form of this specific knowledge later in the article. The next section addresses the question of the socio-genesis of MMR.

Where does mixed methods research come from?

The origins of the subfield of MMR lay in the broader field of social scientific disciplines. We interpret the positions of the scholars most involved in MMR (the “pioneers” or “scientific entrepreneurs”) as occupying particular positions within the larger academic and scientific field. Who, then, are the researchers at the heart of MMR? Leech ( 2010 ) interviewed 4 scholars (out of 6) that she identified as early developers of the field: Alan Bryman (UK; sociology), John Creswell (USA; educational psychology), Jennifer Greene (USA; educational psychology) and Janice Morse (USA; nursing and anthropology). Educated in the 1970s and early 1980s, all four of them indicated that they were initially trained in “quantitative methods” and later acquired skills in “qualitative methods.” For two of them (Bryman and Creswell) the impetus to learn qualitative methods was their involvement in writing on, and teaching of, research methods; for Greene and Morse the initial motivation was more instrumental and related to their concrete research activity at the time. Creswell describes himself as “a postpositivist in the 1970s, self-education as a constructivist through teaching qualitative courses in the 1980s, and advocacy for mixed methods (…) from the 1990s to the present” (Creswell 2011 , p. 269). Of this group, only Morse had the benefit of learning about qualitative methods as part of her educational training (in nursing and anthropology; Leech 2010 , p. 267). Independently, Creswell ( 2012 ) identified (in addition to Bryman, Greene and Morse) John Hunter, Allen Brewer (USA; Northwestern and Boston College) and Nigel Fielding (University of Surrey, UK) as important early movers in MMR.

The selections that Leech and Creswell make regarding the key actors are based on their close involvement with the “MMR movement.” It is corroborated by a simple analysis of the articles that appeared in the Journal of Mixed Methods Research ( JMMR ), founded in 2007 as an outlet for MMR.

Table 1 lists all the authors that have published in the issues of the journal since its first publication in 2007 and that have either received more than 14 (4%) of the citations allocated between the group of 343 authors (the TLCS score in Table 1 ), or have written more than 2 articles for the Journal (1.2% of all the articles that have appeared from 2007 until October 2013) together with their educational background (i.e., the discipline in which they completed their PhD).

All the members of Leech’s selection, except for Morse, and the members of Creswell’s selection (except Hunter, Brewer, and Fielding) are represented in the selection based on the entries in the JMMR . Footnote 5 The same holds for two of the three additional authors identified by Creswell. Hunter and Brewer have developed a somewhat different approach to combining methods that explicitly targets data gathering techniques and largely avoids epistemological discussions. In Brewer and Hunter ( 2006 ) they discuss the MMR approach very briefly and only include two references in their bibliography to the handbook of Tashakkori and Teddlie ( 2003 ), and at the end of 2013 they had not published in the JMMR . Fielding, meanwhile, has written two articles for the JMMR (Fielding and Cisneros-Puebla 2009 ; Fielding 2012 ). In general, it seems reasonable to assume that a publication in a journal that positions itself as part of a systematic attempt to build a research tradition, and can be viewed as part of a strategic effort to advance MMR as a distinct alternative to more “traditional” academic research—particularly in methods—at least signals a degree of adherence to the effort and acceptance of the rules of the game it lays out. This would locate Fielding closer to the MMR movement than the others.

The majority of the researchers listed in Table 1 have a background in psychology or social psychology (35%), and sociology (25%). Most of them work in the United States or are UK citizens, and the positions they occupied at the beginning of 2013 indicates that most of these are in applied research: educational research and educational psychology account for 50% of all the disciplinary occupations of the group that were still employed in academia. This is consistent with the view that MMR originated in applied disciplines and thematic studies like education and nursing, rather than “pure disciplines” like psychology and sociology (Tashakkori and Teddlie ( 2010b ), p. 32). Although most of the 20 individuals mentioned in Table 1 have taught methods courses in academic curricula (for 15 of them, we could determine that they were involved in the teaching of qualitative, quantitative, or mixed methods), there are few individuals with a background in statistics or a neighbouring discipline: only Amy Dellinger did her PhD in “research methodology.” In addition, as far as we could determine, only three individuals held a position in a methodological department at some time: Dellinger, Tony Onwuegbuzie, and Nancy Leech.

The pre-eminence of applied fields in MMR is supported when we turn our attention to the circulation of MMR. To assess this we proceeded as follows. We selected 10 categories in the Web of Science that form a rough representation of the space of social science disciplines, taking care to include the most important so-called “studies.” These thematically orientated, interdisciplinary research areas have progressively expanded since they emerged at the end of the 1960s as a critique of the traditional disciplines (Heilbron et al. 2017 ). For each category, we selected the 10 journals with the highest 5-year impact factor in their category in the period 2007–2015. The lists were compiled bi-annually over this period, resulting in 5 top ten lists for the following Web of Science categories: Economics, Psychology, Sociology, Anthropology, Political Science, Nursing, Education & Educational Research, Business, Cultural Studies, and Family Studies. After removing multiple occurring journals, we obtained a list of 164 journals.

We searched the titles and abstracts of the articles appearing in these journals over the period 1992–2016 for occurrences of the terms “mixed method” or “multiple methods” and variants thereof. We chose this particular period and combination of search terms to see if a shift from a more general use of the term “multiple methods” to “mixed methods” occurred following the institutionalization of MMR. In total, we found 797 articles (out of a total of 241,521 articles that appeared in these journals during that time), published in 95 different journals. Table 2 lists the 20 journals that contain at least 1% (8 articles) of the total amount of articles.

As is clear from Table 2 , the largest number of articles in the sample were published in journals in the field of nursing: 332 articles (42%) appeared in journals that can be assigned to this category. The next largest category is Education & Educational Research, to which 224 (28 percentage) of the articles can be allocated. By contrast, classical social science disciples are barely represented. In Table 2 only the journal Field Methods (Anthropology) and the Journal of Child Psychology and Psychiatry (Psychology) are related to classical disciplines. In Table 3 , the articles in the sample are categorized according to the disciplinary category of the journal in which they appeared. Overall, the traditional disciplines are clearly underrepresented: for the Economics category, for example, only the Journal of Economic Geography contains three articles that make a reference to mixed methods.

Focusing on the core MMR group, the top ten authors of the group together collect 458 citations from the 797 articles in the sample, locating them at the center of the citation network. Creswell is the most cited author (210 citations) and his work too receives most citations from journals in nursing and education studies.

The question whether a terminological shift has occurred from “multiple methods” to “mixed methods” must be answered affirmative for this sample. Prior to 2001 most articles (23 out of 31) refer to “multiple methods” or “multi-method” in their title or abstract, while the term “mixed methods” gains traction after 2001. This shift occurs first in journals in nursing studies, with journals in education studies following somewhat later. The same fields are also the first to cite the first textbooks and handbooks of MMR.

Taken together, these results corroborate the notion that MMR circulates mainly in nursing and education studies. How can this be understood from a field theoretical perspective? MMR can be seen as an innovation in the social scientific field, introducing a new methodology for combining existing methods in research. In general, innovation is a relatively risky strategy. Coming up with a truly rule-breaking innovation often involves a small probability of great success and a large probability of failure. However, it is important to add some nuance to this general observation. First, the risk an innovator faces depends on her position in the field. Agents occupying positions at the top of their field’s hierarchy are rich in specific capital and can more easily afford to undertake risky projects. In the scientific field, these are the agents richest in scientific capital. They have the knowledge, authority, and reputation (derived from recognition by their peers; Bourdieu 2004 , p. 34) that tends to decrease the risk they face and increase the chances of success. Moreover, the positions richest in scientific capital will, by definition, be the most consecrated ones. This consecration involves scientific rather than academic capital (cf. Wacquant 2013 , p. 20) and within disciplines these consecrated positions often are related to orthodox position-takings. This presents a paradox: although they have the capital to take more risks, they have also invested heavily in the orthodoxy of the field and will thus be reluctant to upset the status quo and risk destroying the value of their investment. This results in a tendency to take a more conservative stance, aimed at preserving the status quo in the field and defending their position. Footnote 6

For agents in dominated positions this logic is reversed. Possessing less scientific capital, they hold less consecrated positions and their chances of introducing successful innovations are much lower. This leaves them too with two possible strategies. One is to revert to a strategy of adaptation, accepting the established hierarchy in the field and embarking on a slow advancement to gain the necessary capital to make their mark from within the established order. However, Bourdieu notes that sometimes agents with a relatively marginal position in the field will engage in a “flight forward” and pursue higher risk strategies. Strategies promoting a heterodox approach challenge the orthodoxy and the principles of hierarchization of the field, and, if successful (which will be the case only with a small probability), can rake in significant profits by laying claim to a new orthodoxy (Bourdieu 1975 , p. 104; Bourdieu 1993 , pp. 116–117).

Thus, the coupling of innovative strategies to specific field positions based on the amount of scientific capital alone is not straightforward. It is therefore helpful to introduce a second differentiation in the field that, following Bourdieu ( 1975 , p. 103), is based on the differences between the expected profits from these strategies. Here a distinction can be made between an autonomous and a heteronomous pole of the field, i.e., between the purest, most “disinterested” positions and the most “temporal” positions that are more pervious to the heteronomous logic of social hierarchies outside the scientific field. Of course, this difference is a matter of degree, as even the works produced at the most heteronomous positions still have to adhere to the standards of the scientific field to be seen as legitimate. But within each discipline this dimension captures the difference between agents predominantly engaged in fundamental, scholarly work—“production solely for the producers”—and agents more involved in applied lines of research. The main component of the expected profit from innovation in the first case is scientific, whereas in the second case the balance tends to shift towards more temporal profits. This two-fold structuring of the field allows for a more nuanced conception of innovation than the dichotomy “conservative” versus “radical.” Holders of large amounts of scientific capital at the autonomous pole of the field are the producers and conservators of orthodoxy, producing and diffusing what can be called “orthodox innovations” through their control of relatively powerful networks of consecration and circulation. Innovations can be radical or revolutionary in a rational sense, but they tend to originate from questions raised by the orthodoxy of the field. Likewise, the strategy to innovate in this sense can be very risky in that success is in no way guaranteed, but the risk is mitigated by the assurance of peers that these are legitimate questions, tackled in a way that is consistent with orthodoxy and that does not threaten control of the consecration and circulation networks.

These producers are seen as intellectual leaders by most agents in the field, especially by those aspiring to become part of the specific networks of production and circulation they maintain. The exception are the agents located at the autonomous end of the field who possess less scientific capital and outright reject this orthodoxy produced by the field’s elite. Being strictly focused on the most autonomous principles of legitimacy, they are unable to accommodate and have no choice but to reject the orthodoxy. Their only hope is to engage in heterodox innovations that may one day become the new orthodoxy.

The issue is less antagonistic at the heteronomous side of the field, at least as far as the irreconcilable position-takings at the autonomous pole are concerned. The main battle here is also for scientific capital, but is complemented by the legitimacy it brings to gain access to those who are in power outside of the scientific field. At the dominant side, those with more scientific capital tend to have access to the field of power, agents who hold the most economic and cultural capital, for example by holding positions in policy advisory committees or company boards. The dominated groups at this side of the field will cater more to practitioners or professionals outside of the field of science.

Overall, there will be fewer innovations on this side. Moreover, innovative strategies will be less concerned with the intricacies of the pure discussions that prevail at the autonomous pole and be of a more practical nature, but pursued from different degrees of legitimacy according to the differences in scientific capital. This affects the form these more practical, process-orientated innovations take. At the dominant side of this pole, agents tend to accept the outcome of the struggles at the autonomous pole: they will accept the orthodoxy because mastery of this provides them with scientific capital and the legitimacy they need to gain access to those in power. In contrast, agents at the dominated side will be more interested in doing “what works,” neutralizing the points of conflict at the autonomous pole and deriving less value from strictly following the orthodoxy. This way, a four-fold classification of innovative strategies in the scientific field emerges (see Fig.  2 ) that helps to understand the context in which MMR was developed.

Scientific field and scientific innovation

In summary, the small group of researchers who have been identified as the core of MMR consist predominantly of users of methods, who were educated and have worked exclusively at US and British universities. The specific approach to combining methods that is proposed by MMR has been successful from an institutional point of view, achieving visibility through the foundation of a journal and association and a considerable output of core MMR scholars in terms of books, conference proceedings, and journal articles. Its origins and circulation in vocational studies rather than classical academic disciplines can be understood from the position these studies occupy in the scientific field and the kinds of position-taking and innovations these positions give rise to. This context allows a reflexive understanding of the content of MMR and the issues that are dominant in the approach. We turn to this in the next section.

Mixed methods research: Position-taking

The position of the subfield of MMR in the scientific field is related to the position-takings of agents that form the core of this subfield (Bourdieu 1993 , p. 35). The space of position takings, in turn, provides the framework to study the most salient issues that are debated within the subfield. Since we can consider MMR to be an emerging subfield, where positions and position takings are not as clearly defined as in more mature and settled fields, it comes as no surprise that there is a lively discussion of fundamental matters. Out of the various topics that are actively discussed, we have distilled three themes that are important for the way the subfield of MMR conveys its autonomy as a field and as a distinct approach to research. Footnote 7 In our view, these also represent the main problems with the way MMR approaches the issue of combining methods.

Methodology making and standardization

The first topic is that the approach is moving towards defining a unified MMR methodology. There are differences in opinion as to how this is best achieved, but there is widespread agreement that some kind of common methodological and conceptual foundation of MMR is needed. To this end, some propose a broad methodology that can serve as distinct marker of MMR research. For instance, in their introduction to the handbook, Tashakkori and Teddlie ( 2010b ) propose a definition of the methodology of mixed methods research as “the broad inquiry logic that guides the selection of specific methods and that is informed by conceptual positions common to mixed methods practitioners” (Tashakkori and Teddlie 2010b , p. 5). When they (later on in the text) provide two methodological principles that differentiate MMR from other communities of scholars, they state that they regard it as a “crucial mission” for the MMR community to generate distinct methodological principles (Tashakkori and Teddlie 2010b , pp. 16–17). They envision an MMR methodology that can function as a “guide” for selecting specific methods. Others are more in favour of finding a philosophical foundation that underlies MMR. For instance, Morgan ( 2007 ) and Hesse-Biber ( 2010 ) consider pragmatism as a philosophy that distinguishes MMR from qualitative (constructivism) and quantitative (positivist) research and that can provide a rationale for the paradigmatic pluralism typical of MMR.

Furthermore, there is wide agreement that some unified definition of MMR would be beneficial, but it is precisely here that there is a large variation in interpretations regarding the essentials of MMR. This can be seen in the plethora of definitions that have been proposed. Johnson et al. ( 2007 ) identified 19 alternative definitions of MMR at the time, out of which they condensed their own:

[MMR] is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purpose of breath and depth of understanding and corroboration. Footnote 8

Four years later, the issue is not settled yet. Creswell and Plano Clark ( 2011 ) list a number of authors who have proposed a different definition of MMR, and conclude that there is a common trend in the content of these definitions over time. They take the view that earlier texts on mixing methods stressed a “disentanglement of methods and philosophy,” while later texts locate the practice of mixing methods in “all phases of the research process” (Creswell and Plano Clark 2011 , p. 2). It would seem, then, that according to these authors the definitions of MMR have become more abstract, further away from the practicality of “merely” combining methods. Specifically, researchers now seem to speak of mixing higher order concepts: some speak of mixing methodologies, others refer to mixing “research approaches,” or combining “types of research,” or engage in “multiple ways of seeing the social world” (Creswell and Plano Clark 2011 ).

This shift is in line with the direction in which MMR has developed and that emphasises practical ‘manuals’ and schemas for conducting research. A relatively large portion of the MMR literature is devoted to classifications of mixed methods designs. These classifications provide the basis for typologies that, in turn, provide guidelines to conduct MMR in a concrete research project. Tashakkori and Teddlie ( 2003 ) view these typologies as important elements of the organizational structure and legitimacy of the field. In addition, Leech and Onwuegbuzie ( 2009 ) see typologies as helpful guides for researchers and of pedagogical value (Leech and Onwuegbuzie 2009 , p. 272). Proposals for typologies can be found in textbooks, articles, and contributions to the handbook(s). For example, Creswell et al. ( 2003 , pp. 169-170) reviewed a number of studies and identified 8 different ways to classify MMR studies. This list was updated and extended by Creswell and Plano Clark ( 2011 , pp. 56-59) to 15 typologies. Leech and Onwuegbuzie ( 2009 ) identified 35 different research designs in the contributions to Teddlie and Tashakkori (2003) alone, and proposed their own three-dimensional typology that resulted in 8 different types of mixed methods studies. As another example of the ubiquity of these typologies, Nastasi et al. ( 2010 ) classified a large number of existing typologies in MMR into 7”meta-typologies” that each emphasize different aspects of the research process as important markers for MMR. According to the authors, these typologies have the same function in MMR as the more familiar names of “qualitative” or “quantitative” methods (e.g., “content analysis” or “structural equation modelling”) have: to signal readers of research what is going on, what procedures have been followed, how to interpret results, etc. (see also Creswell et al. 2003 , pp. 162–163). The criteria underlying these typologies mainly have to do with the degree of mixing (e.g., are methods mixed throughout the research project or not?), the timing (e.g., sequential or concurrent mixing of methods) and the emphasis (e.g., is one approach dominant, or do they have equal status?).

We find this strong drive to develop methodologies, definitions, and typologies of MMR as guides to valid mixed methods research problematic. What it amounts to in practice is a methodology that lays out the basic guidelines for doing MMR in a “proper way.” This entails the danger of straight-jacketing reflection about the use of methods, decoupling it from theoretical and empirical considerations, thus favouring the unreflexive use of a standard methodology. Researchers are asked to make a choice for a particular MMR design and adhere to the guidelines for a “proper” MMR study. Such methodological prescription diametrically opposes the initial critique of the mechanical and unreflexive use of methods. The insight offered by Bourdieu’s notion of reflexivity is, on the contrary, that the actual research practice is fundamentally open in terms of being guided by a logic of practice that cannot be captured by a preconceived and all-encompassing logic independent of that practice. Reflexivity in this view cannot be achieved by hiding behind the construct of a standardized methodology—of whatever signature—it can only be achieved by objectifying the process of objectification that goes on within the context of the field in which the researcher is embedded. This reflexivity, then, requires an analysis of the position of the researcher as a critical component of the research process, both as the embodiment of past choices that have consequences for the strategic position in the scientific field, and as predispositions regarding the choice for the subject and content of a research project. By adding the insight of STS researchers that the point of deconstructing science and technology is not so much to offer a new best way of doing science or technology, but to provide insights into the critical moments in research (for a take on such a debate, see, for example, Edge 1995 , pp. 16–20), this calls for a sociology of science that takes methods much more seriously as objects of study. Such a programme should be based on studying the process of codification and standardization of methods in their historical context of production, circulation, and use. It would provide a basis for a sociological understanding of methods that can illuminate the critical moments in research alluded to above, enabling a systematic reflection on the process of objectification. This, in turn, allows a more sophisticated validation of using—and combining—methods than relying on prescribed methodologies.

The role of epistemology

The second theme discussed in a large number of contributions is the role epistemology plays in MMR. In a sense, epistemology provides the lifeblood for MMR in that methods in MMR are mainly seen in epistemological terms. This interpretation of methods is at the core of the knowledge claim of MMR practitioners, i.e., that the mixing of methods means mixing broad, different ways of knowing, which leads to better knowledge of the research object. It is also part of the identity that MMR consciously assumes, and that serves to set it apart from previous, more practical attempts to combine methods. This can be seen in the historical overview that Creswell and Plano Clark ( 2011 ) presented and that was discussed above. This reading, in which combining methods has evolved from the rather unproblematic level (one could alternatively say “naïve” or “unaware”) of instrumental use of various tools and techniques into an act that requires deeper thinking on a methodological and epistemological level, provides the legitimacy of MMR.

At the core of the MMR approach we thus find that methods are seen as unproblematic representations of different epistemologies. But this leads to a paradox, since the epistemological frameworks need to be held flexible enough to allow researchers to integrate elements of each of them (in the shape of methods) into one MMR design. As a consequence, the issue becomes the following: methods need to be disengaged from too strict an interpretation of the epistemological context in which they were developed in order for them to be “mixable,”’, but, at the same time, they must keep the epistemology attributed to them firmly intact.

In the MMR discourse two epistemological positions are identified that matter most: a positivist approach that gives rise to quantitative methods and a constructivist approach that is home to qualitative methods. For MMR to be a feasible endeavour, the differences between both forms of research must be defined as reconcilable. This position necessitates an engagement with those who hold that the quantitative/qualitative dichotomy is unbridgeable. Within MMR an interesting way of doing so has emerged. In the first issue of the Journal of Mixed Methods Research, Morgan ( 2007 ) frames the debate about research methodology in the social sciences in terms of Kuhnian paradigms, and he argues that the pioneers of the emancipation of qualitative research methods used a particular interpretation of the paradigm-concept to state their case against the then dominant paradigm in the social sciences. According to Morgan, they interpreted a paradigm mainly in metaphysical terms, stressing the connections among the trinity of ontology, epistemology, and methodology as used in the philosophy of knowledge (Morgan 2007 , p. 57). This allowed these scholars to depict the line between research traditions in stark, contrasting terms, using Kuhn’s idea of “incommensurability” in the sense of its “early Kuhn” interpretation. This strategy fixed the contrast between the proposed alternative approach (a “constructivist paradigm”), and the traditional approach (constructed as “the positivist paradigm”) to research as a whole, and offered the alternative approach as a valid option rooted in the philosophy of knowledge. Morgan focuses especially on the work of Egon Guba and Yvonne Lincoln who developed what they initially termed a “naturalistic paradigm” as an alternative to their perception of positivism in the social sciences (e.g., Guba and Lincoln 1985 ). Footnote 9 MMR requires a more flexible or “a-paradigmatic stance” towards research, which would entail that “in real-world practice, methods can be separated from the epistemology out of which they emerged” (Patton 2002 , quoted in Tashakkori and Teddlie 2010b , p. 14).

This proposal of an ‘interpretative flexibility’ (Bijker 1987 , 1997 ) regarding paradigms is an interesting proposition. But it immediately raises the question: why stop there? Why not take a deeper look into the epistemological technology of methods themselves, to let the muted components speak up in order to look for alternative “mixing interfaces” that could potentially provide equally valid benefits in terms of the understanding of a research object? The answer, of course, was already seen above. It is that the MMR approach requires situating methods epistemologically in order to keep them intact as unproblematic mediators of specific epistemologies and, thus, make the methodological prescriptions work. There are several problems with this. First, seeing methods solely through an epistemological lens is problematic, but it would be less consequential if it were applied to multiple elements of methods separately. This would at least allow a look under the hood of a method, and new ways of mixing methods could be opened up that go beyond the crude “qualitative” versus “quantitative” dichotomy. Second, there is also the issue of the ontological dimension of methods that is disregarded in an exclusively epistemological framing of methods (e.g., Law 2004 ). Taking this ontological dimension seriously has at least two important facets. First, it draws attention to the ontological assumptions that are woven into methods in their respective fields of production and that are imported into fields of users. Second, it entails the ontological consequences of practising methods: using, applying, and referring to methods and the realities this produces. This latter facet brings the world-making and boundary-drawing capacities of methods to the fore. Both facets are ignored in MMR. We say more about the first facet in the next section. With regard to the second facet, a crucial element concerns the data that are generated, collected, and analysed in a research project. But rather than problematizing the link between the performativity of methods and the data that are enacted within the frame of a method, here too MMR relies on a dichotomy: that between quantitative and qualitative data. Methods are primarily viewed as ways of gathering data or as analytic techniques dealing with a specific kind of data. Methods and data are conceptualised intertwiningly: methods too are seen as either quantitative or qualitative (often written as QUANT and QUAL in the literature), and perform the role of linking epistemology and data. In the final analysis, the MMR approach is based on the epistemological legitimization of the dichotomy between qualitative and quantitative data in order to define and combine methods: data obtain epistemological currency through the supposed in-severable link to certain methods, and methods are reduced to the role of acting as neutral mediators between them.

In this way, methods are effectively reduced to, on the one hand, placeholders for epistemological paradigms and, on the other hand, mediators between one kind of data and the appropriate epistemology. To put it bluntly, the name “mixed methods research” is actually a misnomer, because what is mixed are paradigms or “approaches,” not methods. Thus, the act of mixing methods à la MMR has the paradoxical effect of encouraging a crude black box approach to methods. This is a third problematic characteristic of MMR, because it hinders a detailed study of methods that can lead to a much richer perspective on mixing methods.

Black boxed methods and how to open them

The third problem that we identified with the MMR approach, then, is that with the impetus to standardize the MMR methodology by fixing methods epistemologically, complemented by a dichotomous view of data, they are, in the words of philosopher Bruno Latour, “blackboxed.” This is a peculiar result of the prescription for mixing methods as proposed by MMR that thus not only denies practice and the ontological dimensions of methods and data, but also casts methods in the role of unyielding black boxes. Footnote 10 With this in mind, it will come as no surprise that most foundational contributions to the MMR literature do not explicitly define what a method is, nor that they do not provide an elaborative historical account of individual methods. The particular framing of methods in MMR results in a blind spot for the historical and social context of the production and circulation of methods as intellectual products. Instead it chooses to reify the boundaries that are drawn between “qualitative” and “quantitative” methods and reproduce them in the methodology it proposes. Footnote 11 This is an example of “circulation without context” (Bourdieu 2002 , p. 4): classifications that are constructed in the field of use or reception without taking the constellation within the field of production seriously.

Of course, this does not mean that the reality of the differences between quantitative and qualitative research must be denied. These labels are sticky and symbolically laden. They have come, in many ways, to represent “two cultures” (Goertz and Mahony 2012 ) of research, institutionalised in academia, and the effects of nominally “belonging” to (or being assigned to) one particular category have very real consequences in terms of, for instance, access to research grants and specific journals. However, if the goal of an approach such as MMR is to open up new pathways in social science research, (and why should that not be the case?) it is hard to see how that is accomplished by defining the act of combining methods solely in terms of reified differences between research using qualitative and quantitative data. In our view, methods are far richer and more interesting constructs than that, and a practice of combining methods in research should reflect that. Footnote 12

Addressing these problems entices a reflection on methods and using (multiple) methods that is missing in the MMR perspective. A fruitful way to open up the black boxes and take into account the epistemological and ontological facets of methods is to make them, and their use, the object of sociological-historical investigation. Methods are constituted through particular practices. In Bourdieusian terms, they are objectifications of the subjectively understood practices of scientists “in other fields.” Rather than basing a practice of combining methods on an uncritical acceptance of the historically grown classification of types of social research (and using these as the building stones of a methodology of mixing methods), we propose the development of a multifaceted approach that is based on a study of the different socio-historical contexts and practices in which methods developed and circulated.

A sociological understanding of methods based on these premises provides the tools to break with the dichotomously designed interface for combining methods in MMR. Instead, focusing on the historical and social contexts of production and use can reveal the traces that these contexts leave, both in the internal structure of methods, how they are perceived, how they are put into practice, and how this practice informs the ontological effects of methods. Seeing methods as complex technologies, with a history that entails the struggles among the different agents involved in their production, and use opens the way to identify multiple interfaces for combining them: the one-sided boxes become polyhedra. The critical study of methods as “objects of objectification” also entices analyses of the way in which methods intervene between subject (researcher) and object and the way in which different methods are employed in practice to draw this boundary differently. The reflexive position generated by such a systematic juxtaposition of methods is a fruitful basis to come to a richer perspective on combining methods.

We critically reviewed the emerging practice of combining methods under the label of MMR. MMR challenges the mono-method approaches that are still dominant in the social sciences, and this is both refreshing and important. Combining methods should indeed be taken much more seriously in the social sciences.

However, the direction that the practice of combining methods is taking under the MMR approach seems problematic to us. We identified three main concerns. First, MMR scholars seem to be committed to designing a standardized methodological framework for combining methods. This is unfortunate, since it amounts to enforcing an unnecessary codification of aspects of research practices that should not be formally standardized. Second, MMR constructs methods as unproblematic representations of an epistemology. Although methods must be separable from their native epistemology for MMR to work, at the same time they have to be nested within a qualitative or a quantitative research approach, which are characterized by the data they use. By this logic, combining quantitative methods with other quantitative methods, or qualitative methods with other qualitative methods, cannot offer the same benefits: they originate from the same way of viewing and knowing the world, so it would have the same effect as blending two gradations of the same colour paint. The importance attached to the epistemological grounding of methods and data in MMR also disregards the ontological aspects of methods. In this article, we are arguing that this one-sided perspective is problematic. Seeing combining methods as equivalent to combining epistemologies that are somehow pure and internally homogeneous because they can be placed in a qualitative or quantitative framework essentially amounts to reifying these categories.

It also leads to the third problem: the black boxing of methods as neutral mediators between these epistemologies and data. This not only constitutes a problem for trying to understand methods as intellectual products, but also for regarding the practice of combining methods, because it ignores the social-historical context of the development of individual methods and hinders a sociologically grounded notion of combining methods.

We proceed from a different perspective on methods. In our view, methods are complex constructions. They are world-making technologies that encapsulate different assumptions on causality, rely on different conceptual relations and categorizations, allow for different degrees of emergence, and employ different theories of the data that they internalise as objects of analysis. Even more importantly, their current form as intellectual products cannot be separated from the historical context of their production, circulation, and use.

A fully developed exposition of such an approach will have to await further work. Footnote 13 So far, the sociological study of methods has not (yet) developed into a consistent research programme, but important elements can be derived from existing contributions such as MacKenzie ( 1981 ), Chapoulie ( 1984 ), Platt ( 1996 ), Freeman ( 2004 ), and Desrosières ( 2008a , b ). The work on the “social life of methods” (e.g., Savage 2013 ) also contains important leads for the development of a systematic sociological approach to method production and circulation. Based on the discussion in this article and the contributions listed above, some tantalizing questions can be formulated. How are methods and their elements objectified? How are epistemology and ontology defined in different fields and how do those definitions feed into methods? How do they circulate and how are they translated and used in different contexts? What are the main controversies in fields of users and how are these related to the field of production? What are the homologies between these fields?

Setting out to answer these questions opens up the possibility of exploring other interesting combinations of methods that emerge from the combination of different practices, situated in different historical and epistemological contexts, and with their unique set of interpretations regarding their constituent elements. One of these must surely be the data-theoretical elements that different methods incorporate. The problematization of data has become all the more pressing now that the debate about the consequences of “big data” for social scientific practices has become prominent (Savage and Burrows 2007 ; Levallois et al. 2013 ; Burrows and Savage 2014 ). Whereas MMR emphasizes the dichotomy between qualitative and quantitative data, a historical analysis of the production and use of methods can explore the more subtle, different interpretations and enactments of the “same” data. These differences inform method construction, controversies surrounding methods and, hence, opportunities for combining methods. These could then be constructed based on alternative conceptualisations of data. Again, while in some contexts it might be enlightening to rely on the distinction between data as qualitative or quantitative, and to combine methods based on this categorization, it is an exciting possibility that in other research contexts other conceptualisations of data might be of more value to enhance a specific (contextual) form of knowledge.

Change history

06 may 2019.

Unfortunately, figure 2 was incorrectly published.

The search term used was “mixed method*” in the “topic” search field of SSCI, A&HCI, and CPCI-SSH as contained in the Web of Science. A Google NGram search (not shown) confirmed this pattern. The results of a search for “mixed methods” and “mixed methods research” showed a very steep increase after 1994: in the first case, the normalized share in the total corpus increased by 855% from 1994 till 2008. Also, Creswell ( 2012 ) reports an almost hundred-fold increase in the number of theses and dissertations with mixed methods’ in the citation and abstract (from 26 in 1990–1994 to 2524 in 2005–2009).

Retrieved from https://uk.sagepub.com/en-gb/eur/journal-of-mixed-methods-research/journal201775#aims-and-scope on 1/17/2019.

In terms of antecedents of mixed methods research, it is interesting to note that Bourdieu, whose sociology of science we draw on, was, from his earliest studies in Algeria onwards, a strong advocate of combining research methods. He made it into a central characteristic of his approach to social science in Bourdieu et al. ( 1991 [1968]). His approach, as we see below, was very different from the one now proposed under the banner of MMR. Significantly, there is no mention of Bourdieu’s take on combining methods in any of the sources we studied.

Morse’s example in particular warns us that restricting the analysis to the authors that have published in the JMMR runs the risk of missing some important contributors to the spread of MMR through the social sciences. On her website, Morse lists 11 publications (journal articles, book chapters, and books) that explicitly make reference to mixed methods (and a substantial number of other publications are about methodological aspects of research), so the fact that she has not (yet) published in the JMMR cannot, by itself, be taken as an indication of a lesser involvement with the practice of combining methods. See the website of Janice Morse at https://faculty.utah.edu/u0556920-Janice_Morse_RN,_PhD,_FAAN/hm/index.hml accessed 1/17/2019.

Bourdieu ( 1999 , p. 26) mentions that one has to be a scientific capitalist to be able to start a scientific revolution. But here he refers explicitly to the autonomy of the scientific field, making it virtually impossible for amateurs to stand up against the historically accumulated capital in the field and incite a revolution.

The themes summarize the key issues through which MMR as a group comes “into difference” (Bourdieu 1993 , p. 32). Of course, as in any (sub)field, the agents identified above often differ in their opinions on some of these key issues or disagree on the answer to the question if there should be a high degree of convergence of opinions at all. For instance, Bryman ( 2009 ) worried that MMR could become “a ghetto.” For him, the institutional landmarks of having a journal, conferences, and a handbook increase the risk of “not considering the whole range of possibilities.” He added: “I don’t regard it as a field, I kind of think of it as a way of thinking about how you go about research.” (Bryman, cited in Leech 2010 , p. 261). It is interesting to note that Bryman, like fellow sociologists Morgan and Denscombe, had published only one paper in the JMMR by the end of 2016 (Bryman passed away in June of 2017). Although these papers are among the most cited papers in the journal (see Table 1 ), this low number is consistent with the more eclectic approach that Bryman proposed.

Johnson, Onwuegbuzie, and Turner ( 2007 , p. 123).

Guba and Lincoln ( 1985 ) discuss the features of their version of a positivistic approach mainly in ontological and epistemological terms, but they are also careful to distinguish the opposition between naturalistic and positivist approaches from the difference between what they call the quantitative and the qualitative paradigms. Since they go on to state that, in principle, quantitative methods can be used within a naturalistic approach (although in practice, qualitative methods would be preferred by researchers embracing this paradigm), they seem to locate methods on a somewhat “lower,” i.e., less incommensurable level. However, in their later work (both together as well as with others or individually) and that of others in their wake, there seems to have been a shift towards a stricter interpretation of the qualitative/quantitative divide in metaphysical terms, enabling Teddlie and Tashakkori (2010b) to label this group “purists” (Tashakkori and Teddlie 2010b , p. 13).

See, for instance, Onwuegbuzie et al.’s ( 2011 ) classification of 58 qualitative data analysis techniques and 18 quantitative data analysis techniques.

This can also be seen in Morgan’s ( 2018 ) response to Sandelowski’s ( 2014 ) critique of the binary distinctions in MMR between qualitative and quantitative research approaches and methods. Morgan denounces the essentialist approach to categorizing qualitative and quantitative research in favor of a categorization based on “family resemblances,” in which he draws on Wittgenstein. However, this denies the fact that the essentialist way of categorizing is very common in the MMR corpus, particularly in textbooks and manuals (e.g., Plano Clark and Ivankova 2016 ). Moreover, and more importantly, he still does not extend this non-essentialist model of categorization to the level of methods, referring, for instance, to the different strengths of qualitative and quantitative methods in mixed methods studies (Morgan 2018 , p. 276).

While it goes beyond the scope of this article to delve into the history of the qualitative-quantitative divide in the social sciences, some broad observations can be made here. The history of method use in the social sciences can briefly be summarized as first, a rather fluid use of what can retrospectively be called different methods in large scale research projects—such as the Yankee City study of Lloyd Warner and his associates (see Platt 1996 , p. 102), the study on union democracy of Lipset et al. ( 1956 ), and the Marienthal study by Lazarsfeld and his associates (Jahoda et al. 1933 ); see Brewer and Hunter ( 2006 , p. xvi)—followed by an increasing emphasis on quantitative data and the objectification and standardization of methods. The rise of research using qualitative data can be understood as a reaction against this use and interpretation of method in the social sciences. However, out of the ensuing clash a new, still dominant classification of methods emerged, one that relies on the framing of methods as either “qualitative” or “quantitative.” Moreover, these labels have become synonymous with epistemological positions that are reproduced in MMR.

A proposal to come to such an approach can be found in Timans ( 2015 ).

Bijker, W. (1987). The social construction of bakelite: Toward a theory of invention. In W. Bijker, T. Hughes, T. Pinch, & D. Douglas (Eds.), The social construction of technological systems: New directions in the sociology and history of technology . Cambridge, MA: MIT press.

Google Scholar  

Bijker, W. (1997). Of bicycles, bakelites, and bulbs: Toward a theory of sociotechnical change . Cambridge, MA: MIT Press.

Bourdieu, P. (1975). La spécifité du champ scientifique et les conditions sociales du progrès de la raison. Sociologie et Sociétés, 7 (1), 91–118.

Article   Google Scholar  

Bourdieu, P. (1988). Homo academicus . Stanford, CA: Stanford University Press.

Bourdieu, P. (1993). The field of cultural production . Cambridge, UK: Polity Press.

Bourdieu, P. (1999). Les historiens et la sociologie de Pierre Bourdieu. Le Bulletin de la Société d'Histoire Moderne et Contemporaine/SHMC, 1999 (3&4), 4–27.

Bourdieu, P. (2002). Les conditions sociales de la circulation internationale des idées. Actes de la Recherche en Sciences Sociales, 145 (5), 3–8.

Bourdieu, P. (2004). Science of science and reflexivity . Cambridge, UK: Polity.

Bourdieu, P. (2007). Sketch for a self-analysis . Cambridge, UK: Polity.

Bourdieu, P., Chamboredon, J., & Passeron, J. (1991). The craft of sociology: Epistemological preliminaries . Berlin, Germany: De Gruyter.

Book   Google Scholar  

Brewer, J., & Hunter, A. (2006). Multimethod research: A synthesis of styles . London, UK: Sage.

Bryman, A. (2009). Sage Methodspace: Alan Bryman on research methods. Retrieved from http://www.youtube.com/watch?v=bHzM9RlO6j0 . Accessed 3/7/2019.

Bryman, A. (2012). Social research methods . Oxford, UK: Oxford University Press.

Burrows, R., & Savage, M. (2014). After the crisis? Big data and the methodological challenges of empirical sociology. Big Data & Society, 1 (1), 1–6.

Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56 (2), 81–105.

Chapoulie, J. (1984). Everett C. Hughes et le développement du travail de terrain en sociologie. Revue Française de Sociologie, 25 (4), 582–608.

Collins, R. (1998). The sociology of philosophies: A global theory of intellectual change . Cambridge, MA: Harvard University Press.

Creswell, J. (2008). Research design: Qualitative, quantitative, and mixed methods approaches . Thousand Oaks, CA: Sage.

Creswell, J. (2011). Controversies in mixed methods research. In N. Denzin & Y. Lincoln (Eds.), The Sage handbook of qualitative research . Thousand Oaks, CA: Sage.

Creswell, J. (2012). Qualitative inquiry and research design: Choosing among five approaches . Thousand Oaks, CA: Sage.

Creswell, J., & Plano Clark, V. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

Creswell, J., Plano Clark, V., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Denscombe, M. (2008). Communities of practice a research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2 (3), 270–283.

Desrosières, A. (2008a). Pour une sociologie historique de la quantification - L’Argument statistique I . Paris, France: Presses des Mines.

Desrosières, A. (2008b). Gouverner par les nombres - L’Argument statistique II . Paris, France: Presses des Mines.

Edge, D. (1995). Reinventing the wheel. In D. Edge, S. Jasanof, G. Markle, J. Petersen, & T. Pinch (Eds.), Handbook of science and technology studies . Thousand Oaks, CA: Sage.

Fielding, N. (2012). Triangulation and mixed methods designs data integration with new research technologies. Journal of Mixed Methods Research, 6 (2), 124–136.

Fielding, N., & Cisneros-Puebla, C. (2009). CAQDAS-GIS convergence: Toward a new integrated mixed method research practice? Journal of Mixed Methods Research, 3 (4), 349–370.

Fleck, C., Heilbron, J., Karady, V., & Sapiro, G. (2016). Handbook of indicators of institutionalization of academic disciplines in SSH. Serendipities, Journal for the Sociology and History of the Social Sciences, 1 (1) Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/1 . Accessed 10/10/2018.

Freeman, L. (2004). The development of social network analysis: A study in the sociology of science . Vancouver, Canada: Empirical Press.

Gingras, Y., & Gemme, B. (2006). L’Emprise du champ scientifique sur le champ universitaire et ses effets. Actes de la Recherche en Sciences Sociales, 164 , 51–60.

Goertz, G., & Mahony, J. (2012). A tale of two cultures: Qualitative and quantitative research in the social sciences . Princeton: Princeton University Press.

Greene, J. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2 (1), 7–22.

Guba, E., & Lincoln, Y. (1985). Naturalistic inquiry . Thousand Oaks, CA: Sage.

Heilbron, J., Bedecarré, M., & Timans, R. (2017). European journals in the social sciences and humanities. Serendipities, Journal for the Sociology and History of the Social Sciences, 2 (1), 33–49 Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/5 . Accessed 10/10/2018.

Hesse-Biber, S. (2010). Mixed methods research: Merging theory with practice . New York, NY: Guilford Press.

Jahoda, M., Lazarsfeld, P., & Zeisel, H. (1933). Die Arbeitslosen von Marienthal. Psychologische Monographen, 5 .

Johnson, R., & Gray, R. (2010). A history of philosophical and theoretical issues for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1 (2), 112–133.

Law, J. (2004). After method: Mess in social science research . London, UK: Routledge.

Leech, N. (2010). Interviews with the early developers of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Leech, N., & Onwuegbuzie, T. (2009). A typology of mixed methods research designs. Quality & Quantity, 43 (2), 265–275.

Levallois, C., Steinmetz, S., & Wouters, P. (2013). Sloppy data floods or precise social science methodologies? In P. Wouters, A. Beaulieu, A. Scharnhorst, & S. Wyatt (Eds.), Virtual knowledge . Cambridge, MA: MIT Press.

Lipset, S., Trow, M., & Coleman, J. (1956). Union democracy: The internal politics of the international typographical union . Glencoe, UK: Free Press.

MacKenzie, D. (1981). Statistics in Britain: 1865–1930: The social construction of scientific knowledge . Edinburgh, UK: Edinburgh University Press.

Morgan, D. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1 (1), 48–76.

Morgan, D. (2018). Living with blurry boundaries: The values of distinguishing between qualitative and quantitative research. Journal of Mixed Methods Research, 12 (3), 268–276.

Mullins, N. (1973). Theories and theory groups in contemporary American sociology . New York, NY: Harper & Row.

Nagy Hesse-Biber, S., & Leavy, P. (Eds.). (2008). Handbook of emergent methods . New York, NY and London, UK: Guilford Press.

Nastasi, B., Hitchcock, J., & Brown, L. (2010). An inclusive framework for conceptualizing mixed method design typologies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Onwuegbuzie, A., Leech, N., & Collins, K. (2011). Toward a new era for conducting mixed analyses: The role of quantitative dominant and qualitative dominant crossover mixed analyses. In M. Williams & P. Vogt (Eds.), Handbook of innovation in social research methods . London, UK: Sage.

Patton, M. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

Plano Clark, V., & Creswell, J. (Eds.). (2007). The mixed methods reader . Thousand Oaks, CA: Sage.

Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field . Thousand Oaks, CA: Sage.

Platt, J. (1996). A history of sociological research methods in America: 1920–1960 . Cambridge, UK: Cambridge University Press.

Plowright, D. (2010). Using mixed methods – Frameworks for an integrated methodology . Thousand Oaks, CA: Sage.

Sandelowski, M. (2014). Unmixing mixed methods research. Research in Nursing & Health, 37 (1), 3–8.

Sapiro, G., Brun, E., & Fordant, C. (2018). The rise of the social sciences and humanities in France: Iinstitutionalization, professionalization and autonomization. In C. Fleck, M. Duller, & V. Karady (Eds.), Shaping human science disciplines: Institutional developments in Europe and beyond . Basingstoke, UK: Palgrave.

Savage, M. (2013). The ‘social life of methods’: A critical introduction. Theory, Culture and Society, 30 (4), 3–21.

Savage, M., & Burrows, R. (2007). The coming crisis of empirical sociology. Sociology, 41 (5), 885–899.

Steinmetz, G. (2016). Social fields, subfields and social spaces at the scale of empires: Explaining the colonial state and colonial sociology. The Sociological Review, 64(2_suppl). 98-123.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (Eds.). (2010a). Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2010b). Overview of contemporary issues in mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Chapter   Google Scholar  

Tashakkori, A., & Teddlie, C. (2010c). Epilogue. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2011). Mixed methods research: Contemporary issues in an emerging field. In N. Denzin & Y. Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed.). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2008). Foundations of mixed methods research – Integrating quantitative and qualitative approaches in the social and behavioral sciences . Thousand Oaks, CA: Sage.

Timans, R. (2015). Studying the Dutch business elite: Relational concepts and methods . Doctoral dissertation, Erasmus University Rotterdam, the Netherlands.

Wacquant, L. (2013). Bourdieu 1993: A case study in scientific consecration. Sociology, 47 (1), 15–29.

Download references

Acknowledgments

This research is part of the Interco-SSH project, funded by the European Union under the 7th Research Framework Programme (grant agreement no. 319974). Johan Heilbron would like to thank Louise and John Steffens, members of the Friends Founders’ Circle, who assisted his stay at the Princeton Institute for Advanced Study in 2017-18 during which he completed his part of the present article.

Author information

Authors and affiliations.

Erasmus Centre for Economic Sociology (ECES), Erasmus University Rotterdam, Rotterdam, Netherlands

Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, Netherlands

Paul Wouters

Erasmus Centre for Economic Sociology (ECES), Rotterdam and Centre européen de sociologie et de science politique de la Sorbonne (CESSP-CNRS-EHESS), Paris, France

Johan Heilbron

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Timans .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Timans, R., Wouters, P. & Heilbron, J. Mixed methods research: what it is and what it could be. Theor Soc 48 , 193–216 (2019). https://doi.org/10.1007/s11186-019-09345-5

Download citation

Published : 29 March 2019

Issue Date : 01 April 2019

DOI : https://doi.org/10.1007/s11186-019-09345-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data, field analysis
• Mixed methods research
• Multiple methods
• Reflexivity
• Sociology of science
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 15 December 2015

Qualitative and mixed methods in systematic reviews

• David Gough 1  

Systematic Reviews volume  4 , Article number:  181 ( 2015 ) Cite this article

23k Accesses

42 Citations

23 Altmetric

Metrics details

Expanding the range of methods of systematic review

The logic of systematic reviews is very simple. We use transparent rigorous approaches to undertake primary research, and so we should do the same in bringing together studies to describe what has been studied (a research map) or to integrate the findings of the different studies to answer a research question (a research synthesis). We should not really need to use the term ‘systematic’ as it should be assumed that researchers are using and reporting systematic methods in all of their research, whether primary or secondary. Despite the universality of this logic, systematic reviews (maps and syntheses) are much better known in health research and for answering questions of the effectiveness of interventions (what works). Systematic reviews addressing other sorts of questions have been around for many years, as in, for example, meta ethnography [ 1 ] and other forms of conceptual synthesis [ 2 ], but only recently has there been a major increase in the use of systematic review approaches to answer other sorts of research questions.

There are probably several reasons for this broadening of approach. One may be that the increased awareness of systematic reviews has made people consider the possibilities for all areas of research. A second related factor may be that more training and funding resources have become available and increased the capacity to undertake such varied review work.

A third reason could be that some of the initial anxieties about systematic reviews have subsided. Initially, there were concerns that their use was being promoted by a new managerialism where reviews, particularly effectiveness reviews, were being used to promote particular ideological and theoretical assumptions and to indirectly control research agendas. However, others like me believe that explicit methods should be used to enable transparency of perspectives driving research and to open up access to and participation in research agendas and priority setting [ 3 ] as illustrated, for example, by the James Lind Alliance (see http://www.jla.nihr.ac.uk/ ).

A fourth possible reason for the development of new approaches is that effectiveness reviews have themselves broadened. Some ‘what works’ reviews can be open to criticism for only testing a ‘black box’ hypothesis of what works with little theorizing or any logic model about why any such hypothesis should be true and the mechanisms involved in such processes. There is now more concern to develop theory and to test how variables combine and interact. In primary research, qualitative strategies are advised prior to undertaking experimental trials [ 4 , 5 ] and similar approaches are being advocated to address complexity in reviews [ 6 ], in order to ask questions and use methods that address theories and processes that enable an understanding of both impact and context.

This Special Issue of Systematic Reviews Journal is providing a focus for these new methods of review whether these use qualitative review methods on their own or mixed together with more quantitative approaches. We are linking together with the sister journal Trials for this Special Issue as there is a similar interest in what qualitative approaches can and should contribute to primary research using experimentally controlled trials (see Trials Special Issue editorial by Claire Snowdon).

Dimensions of difference in reviews

Developing the range of methods to address different questions for review creates a challenge in describing and understanding such methods. There are many names and brands for the new methods which may or may not withstand the changes of historical time, but another way to comprehend the changes and new developments is to consider the dimensions on which the approaches to review differ [ 7 , 8 ].

One important distinction is the research question being asked and the associated paradigm underlying the method used to address this question. Research assumes a particular theoretical position and then gathers data within this conceptual lens. In some cases, this is a very specific hypothesis that is then tested empirically, and sometimes, the research is more exploratory and iterative with concepts being emergent and constructed during the research process. This distinction is often labelled as quantitative or positivist versus qualitative or constructionist. However, this can be confusing as much research taking a ‘quantitative’ perspective does not have the necessary numeric data to analyse. Even if it does have such data, this might be explored for emergent properties. Similarly, research taking a ‘qualitative’ perspective may include implicit quantitative themes in terms of the extent of different qualitative findings reported by a study.

Sandelowski and colleagues’ solution is to consider the analytic activity and whether this aggregates (adds up) or configures (arranges) the data [ 9 ]. In a randomized controlled trial and an effectiveness review of such studies, the main analysis is the aggregation of data using a priori non-emergent strategies with little iteration. However, there may also be post hoc analysis that is more exploratory in arranging (configuring) data to identify patterns as in, for example, meta regression or qualitative comparative analysis aiming to identify the active ingredients of effective interventions [ 10 ]. Similarly, qualitative primary research or reviews of such research are predominantly exploring emergent patterns and developing concepts iteratively, yet there may be some aggregation of data to make statements of generalizations of extent.

Even where the analysis is predominantly configuration, there can be a wide variation in the dimensions of difference of iteration of theories and concepts. In thematic synthesis [ 11 ], there may be few presumptions about the concepts that will be configured. In meta ethnography which can be richer in theory, there may be theoretical assumptions underlying the review question framing the analysis. In framework synthesis, there is an explicit conceptual framework that is iteratively developed and changed through the review process [ 12 , 13 ].

In addition to the variation in question, degree of configuration, complexity of theory, and iteration are many other dimensions of difference between reviews. Some of these differences follow on from the research questions being asked and the research paradigm being used such as in the approach to searching (exhaustive or based on exploration or saturation) and the appraisal of the quality and relevance of included studies (based more on risk of bias or more on meaning). Others include the extent that reviews have a broad question, depth of analysis, and the extent of resultant ‘work done’ in terms of progressing a field of inquiry [ 7 , 8 ].

Mixed methods reviews

As one reason for the growth in qualitative synthesis is what they can add to quantitative reviews, it is not surprising that there is also growing interest in mixed methods reviews. This reflects similar developments in primary research in mixing methods to examine the relationship between theory and empirical data which is of course the cornerstone of much research. But, both primary and secondary mixed methods research also face similar challenges in examining complex questions at different levels of analysis and of combining research findings investigated in different ways and may be based on very different epistemological assumptions [ 14 , 15 ].

Some mixed methods approaches are convergent in that they integrate different data and methods of analysis together at the same time [ 16 , 17 ]. Convergent systematic reviews could be described as having broad inclusion criteria (or two or more different sets of criteria) for methods of primary studies and have special methods for the synthesis of the resultant variation in data. Other reviews (and also primary mixed methods studies) are sequences of sub-reviews in that one sub-study using one research paradigm is followed by another sub-study with a different research paradigm. In other words, a qualitative synthesis might be used to explore the findings of a prior quantitative synthesis or vice versa [ 16 , 17 ].

An example of a predominantly aggregative sub-review followed by a configuring sub-review is the EPPI-Centre’s mixed methods review of barriers to healthy eating [ 18 ]. A sub-review on the effectiveness of public health interventions showed a modest effect size. A configuring review of studies of children and young people’s understanding and views about eating provided evidence that the public health interventions did not take good account of such user views research, and that the interventions most closely aligned to the user views were the most effective. The already mentioned qualitative comparative analysis to identify the active ingredients within interventions leading to impact could also be considered a qualitative configuring investigation of an existing quantitative aggregative review [ 10 ].

An example of a predominantly configurative review followed by an aggregative review is realist synthesis. Realist reviews examine the evidence in support of mid-range theories [ 19 ] with a first stage of a configuring review of what is proposed by the theory or proposal (what would need to be in place and what casual pathways would have to be effective for the outcomes proposed by the theory to be supported?) and a second stage searching for empirical evidence to test for those necessary conditions and effectiveness of the pathways. The empirical testing does not however use a standard ‘what works’ a priori methods approach but rather a more iterative seeking out of evidence that confirms or undermines the theory being evaluated [ 20 ].

Although sequential mixed methods approaches are considered to be sub-parts of one larger study, they could be separate studies as part of a long-term strategic approach to studying an issue. We tend to see both primary studies and reviews as one-off events, yet reviews are a way of examining what we know and what more we want to know as a strategic approach to studying an issue over time. If we are in favour of mixing paradigms of research to enable multiple levels and perspectives and mixing of theory development and empirical evaluation, then we are really seeking mixed methods research strategies rather than simply mixed methods studies and reviews.

Noblit G. Hare RD: meta-ethnography: synthesizing qualitative studies. Newbury Park NY: Sage Publications; 1988.

Google Scholar  

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59.

Article   PubMed   PubMed Central   Google Scholar  

Gough D, Elbourne D. Systematic research synthesis to inform policy, practice and democratic debate. Soc Pol Soc. 2002;2002:1.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance 2015. BMJ. 2015;350:h1258

Candy B, Jone L, King M, Oliver S. Using qualitative evidence to help understand complex palliative care interventions: a novel evidence synthesis approach. BMJ Support Palliat Care. 2014;4:Supp A41–A42.

Article   Google Scholar  

Noyes J, Gough D, Lewin S, Mayhew A, Michie S, Pantoja T, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66:11.

Gough D, Oliver S, Thomas J. Introduction to systematic reviews. London: Sage; 2012.

Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28.

Sandelowski M, Voils CJ, Leeman J, Crandlee JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012;6:4.

Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3:67.

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.

Oliver S, Rees R, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Exp. 2008;11:72–84.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Qual Saf. 2015. 2014-003642.

Brannen J. Mixed methods research: a discussion paper. NCRM Methods Review Papers, 2006. NCRM/005.

Creswell J. Mapping the developing landscape of mixed methods research. In: Teddlie C, Tashakkori A, editors. SAGE handbook of mixed methods in social & behavioral research. New York: Sage; 2011.

Morse JM. Principles of mixed method and multi-method research design. In: Teddlie C, Tashakkori A, editors. Handbook of mixed methods in social and behavioural research. London: Sage; 2003.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45.

Harden A, Thomas J. Mixed methods and systematic reviews: examples and emerging issues. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in the social and behavioral sciences. 2nd ed. London: Sage; 2010. p. 749–74.

Chapter   Google Scholar  

Pawson R. Evidenced-based policy: a realist perspective. London: Sage; 2006.

Book   Google Scholar  

Gough D. Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Med. 2013;11:22.

Download references

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, University College London, London, WC1H 0NR, UK

David Gough

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to David Gough .

Additional information

Competing interests.

The author is a writer and researcher in this area. The author declares that he has no other competing interests.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gough, D. Qualitative and mixed methods in systematic reviews. Syst Rev 4 , 181 (2015). https://doi.org/10.1186/s13643-015-0151-y

Download citation

Received : 13 October 2015

Accepted : 29 October 2015

Published : 15 December 2015

DOI : https://doi.org/10.1186/s13643-015-0151-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Subscribe to journal Subscribe
• Get new issue alerts Get alerts
• Submit your manuscript

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

Methodological guidance for the conduct of mixed methods systematic reviews

Stern, Cindy 1,∗ ; Lizarondo, Lucylynn 1,∗ ; Carrier, Judith 2,3 ; Godfrey, Christina 4,5 ; Rieger, Kendra 6 ; Salmond, Susan 7,8 ; Apostolo, João 9,10 ; Kirkpatrick, Pamela 11,12 ; Loveday, Heather 13,14

1 JBI, Faculty of Health and Medical Sciences, The University of Adelaide, Adelaide, SA, Australia

2 School of Healthcare Sciences, Cardiff University, Cardiff, Wales

3 The Wales Centre For Evidence Based Care: A JBI Centre of Excellence

4 School of Nursing, Queen's University, Kingston, ON, Canada

5 Queen's Collaboration for Health Care Quality: A JBI Centre of Excellence, Kingston, ON, Canada

6 College of Nursing, University of Manitoba, Winnipeg, MB, Canada

7 School of Nursing, Rutgers, The State University of New Jersey, New Brunswick, NJ, USA

8 The Northeast Institute for Evidence Synthesis and Translation (NEST): A JBI Centre of Excellence, Newark, NJ, USA

9 Escola Superior de Enfermagem de Coimbra, Coimbra, Portugal

10 Portugal Centre for Evidence Based Practice: A JBI Centre of Excellence, Coimbra, Portugal

11 The Institute for Health and Wellbeing Research, Robert Gordon University, Aberdeen, Scotland

12 The Scottish Centre for Evidence-based, Multi-professional Practice: A JBI Centre of Excellence, Aberdeen, Scotland

13 College of Nursing, Midwifery and Healthcare, University of West London, London, England

14 The University of West London Centre for Evidence-Based Healthcare: A JBI Affiliated Group, London, UK

Correspondence: Cindy Stern, [email protected]

This article was first published in JBI Evid Synth 2020;18:2108–2118.

CS and JC are Senior Associate Editors of JBI Evidence Synthesis . CG is an Associate Editor of JBI Evidence Synthesis . SS is a member of the Editorial board of JBI Evidence Synthesis .

All authors are members of the JBI Mixed Methods Review Methodology Group.

Objective: 

The objective of this paper is to outline the updated methodological approach for conducting a JBI mixed methods systematic review with a focus on data synthesis, specifically, methods related to how data are combined and the overall integration of the quantitative and qualitative evidence.

Introduction: 

Mixed methods systematic reviews provide a more complete basis for complex decision-making than that currently offered by single method reviews, thereby maximizing their usefulness to clinical and policy decision-makers. Although mixed methods systematic reviews are gaining traction, guidance regarding the methodology of combining quantitative and qualitative data is limited. In 2014, the JBI Mixed Methods Review Methodology Group developed guidance for mixed methods systematic reviews; however, since the introduction of this guidance, there have been significant developments in mixed methods synthesis. As such, the methodology group recognized the need to revise the guidance to align it with the current state of knowledge on evidence synthesis methodology

Methods: 

Between 2015 and 2019, the JBI Mixed Methods Review Methodology Group undertook an extensive review of the literature, held annual face-to-face meetings (which were supplemented by teleconferences and regular email correspondence), sought advice from experts in the field, and presented at scientific conferences. This process led to the development of guidance in the form of a chapter in the JBI Manual for Evidence Synthesis , the official guidance for conducting JBI systematic reviews. In 2019, the guidance was ratified by the JBI International Scientific Committee.

Results: 

The updated JBI methodological guidance for conducting a mixed methods systematic review recommends that reviewers take a convergent approach to synthesis and integration whereby the specific method utilized is dependent on the nature/type of questions that are posed in the systematic review. The JBI guidance is primarily based on Hong et al. and Sandelowski's typology on mixed methods systematic reviews. If the review question can be addressed by both quantitative and qualitative research designs, the convergent integrated approach should be followed, which involves data transformation and allows reviewers to combine quantitative and qualitative data. If the focus of the review is on different aspects or dimensions of a particular phenomenon of interest, the convergent segregated approach is undertaken, which involves independent synthesis of quantitative and qualitative data leading to the generation of quantitative and qualitative evidence, which are then integrated together.

Conclusions: 

The updated guidance on JBI mixed methods systematic reviews provides foundational work to a rapidly evolving methodology and aligns with other seminal work undertaken in the field of mixed methods synthesis. Limitations to the current guidance are acknowledged, and a series of methodological projects identified by the JBI Mixed Methods Review Methodology Group to further refine the methodology are proposed. Mixed methods reviews offer an innovative framework for generating unique insights related to the complexities associated with health care quality and safety.

Introduction

Qualitative and quantitative systematic reviews each contribute to our understanding of the best available evidence on a topic, yet increasingly, both perspectives are required to inform clinical policy or organizational decisions. Decision-makers who use systematic reviews have argued for a more complete synthesis of the evidence than that currently offered by single method reviews. 1 Mixed methods systematic reviews (MMSRs) have become an important development in evidence-based health care as they maximize the ability of review findings to assist in clinical and policy decision-making. This type of review is also referred to as mixed methods research syntheses 2 or mixed research syntheses. 3

The conceptual foundation of MMSRs is informed by two research paradigms, namely positivism and constructivism. Positivism is associated with quantitative studies, such as prevalence/incidence or descriptive studies, or an analytical study that examines associations between variables or a cause-and-effect relationship. 4 Conversely, constructivism is commonly associated with qualitative studies that explore a complex phenomenon of interest. 4 Through the development of well-structured MMSRs, the objective numerical data inherent in the logical positivist paradigm combines with the equally important subjective opinions and perspectives presented in the constructivist paradigm. For example, Classen and Lopez 5 used a mixed methods review approach to achieve a better understanding of safety issues among older drivers. An initial quantitative synthesis identified risk and protective factors of older driver safety (i.e., etiologic studies), followed by a synthesis of qualitative studies that captured the perspectives of older adults relating to their driving ability and safety. 5 Without the integration of quantitative results and qualitative results, a complete overarching picture of the inherent complexities associated with older driver safety could not be obtained.

More commonly, MMSRs bring together the findings of effectiveness (quantitative evidence) and patient experiences (qualitative evidence) to allow better understanding of whether and how an intervention works and inform subsequent clinical decision making. For example, although quantitative evidence suggests that the use of larval therapy is clinically and financially effective in the debridement of wounds, 6-10 evidence from qualitative studies indicates that negative patient experiences and perceptions impact the acceptability of the therapy. 11,12 Much like the previous example, without “combining the power of stories and the power of numbers,” 4 (p.29) the understanding about the treatment of wounds using larval therapy is incomplete, which can preclude the development of best-practice recommendations.

Depending on the review question posed, MMSRs can examine the degree of concordance between quantitative and qualitative data to validate or triangulate results and findings, identify discrepancies within the available evidence, and determine whether the quantitative and qualitative data address different aspects of a phenomenon of interest (which can subsequently assist in highlighting gaps in research). Mixed methods systematic reviews also allow one type of data to explore, contextualize, or explain the findings of the other type of data. The methodology for conducting MMSRs is an emerging field of enquiry. Although there is a degree of complexity in conducting MMSRs, the core intention is to combine quantitative and qualitative data (from primary studies) or integrate quantitative and qualitative evidence to create a breadth and depth of understanding that can confirm or dispute evidence and ultimately answer the review question posed. Although MMSRs are gaining traction among health care professionals due to their usefulness and practicality, guidance regarding the methodology of combining quantitative and qualitative data is limited and largely at the theoretical stage. 13-21

In 2014, the JBI Mixed Methods Review Methodology Group developed guidance for MMSRs based on the segregated approach to mixed methods synthesis as described by Sandelowski et al. , 3 which consists of separate syntheses of the quantitative and qualitative components of the systematic review. 14,22 A Bayesian approach was then recommended to pool the findings from the individual syntheses. Since the introduction of this guidance, there have been significant developments in the area of mixed methods synthesis. 13,15,17,23-25 As such, the methodology group recognized the need to revise the guidance to ensure it was accurate and aligned with the current evidence base.

This article describes the methods utilized to revise the guidance and presents the updated methodological approach for undertaking such reviews. It focuses on the conduct of MMSRs as opposed to the reporting of MMSRs; the full official guidance (including reporting requirements) is available in the JBI Manual for Evidence Synthesis . 26 Mixed methods systematic reviews share features that apply to all types of reviews, including formulation of review questions, establishment of eligibility criteria, development of a search strategy, searching and retrieval of relevant studies, assessment of methodological quality, and data extraction. Therefore, the focus of this paper is on illustrating the distinct features of MMSRs as they relate to data synthesis, specifically, methods related to how data are combined and the overall integration of the quantitative and qualitative evidence.

In 2015, it became apparent to the JBI Mixed Methods Review Methodology Group that revision of the guidance was required. In the following year, the group convened to revisit the existing guidance and update the MMSR methodology. The group was composed of a chair (responsible for chairing the meetings and providing feedback on written work), two convenors (responsible for drafting and coordination of written work, organizing meetings, and reporting progress to the JBI Scientific Committee), and six members (responsible for attending meetings regularly and providing feedback on written work). All members were academics and experienced in conducting different types of systematic reviews. Group members were from Australia, Canada, Portugal, the United Kingdom, and the United States of America. An extensive review of the literature was undertaken, which focused on locating all available methodological guidance in the area of MMSRs as well as published examples of MMSRs. Where required, other experts in the field of mixed methods synthesis were contacted for support and clarification. A series of teleconferences and annual face-to-face meetings were also held between 2016 and 2018, and were supplemented by regular email correspondence. Half-day face-to-face meetings were held on the following dates: November 10, 2016 (Adelaide, South Australia), September 15, 2017 (Cape Town, South Africa), and May 1, 2018 (Antwerp, Belgium). Minutes were recorded to ensure a formal approach to tracking progress, allocating work and responsibilities, and completing milestones was maintained. The proposed guidance was presented at scientific conferences in South Africa (2017 Global Evidence Summit) and Belgium (2018 10 th Biennial JBI Colloquium), during which international researchers provided comments that were valuable in informing the methodology.

The final draft of the updated guidance (as a chapter in the JBI Manual for Evidence Synthesis ) was completed following a consensus among members, and on August 6, 2018, was submitted to the JBI International Scientific Committee for consideration, discussion, and approval. Following initial submission, the committee approved the guidance pending minor revisions. Comments and feedback were formally addressed by the methodology group, and a revised version was resubmitted on January 31, 2019. On February 13, 2019, the JBI MMSR methodological guidance was ratified at a meeting of the Scientific Committee and thus supersedes all previous MMSR guidance produced by JBI. 14,22

Results: JBI methodological approach for conducting an MMSR

To avoid confusion in describing this approach, it is important to outline the core concepts related to MMSRs to fully inform this approach ( Table 1 ).

The JBI approach to MMSRs is based on the typology developed by Hong et al. 17 in a review of systematic reviews, which examined the different methods used to synthesize quantitative and qualitative data or integrate quantitative and qualitative evidence. Following the inclusion of 459 reviews, Hong and colleagues 17 identified a number of frameworks used for integration. However, in their work, it became evident there were two frameworks that were predominant: the convergent approach (where the synthesis occurs simultaneously) and the sequential approach (where the synthesis occurs consecutively). 17 Based on minimal usage of the sequential approach by systematic reviewers (approximately 5%), 17 the JBI MMSR methodology currently focuses exclusively on the convergent approach. The convergent design can be broken down into a series of methods that have been simplified into two groups: convergent integrated (which involves data transformation and allows reviewers to combine quantitative and qualitative data), and convergent segregated (which involves independent synthesis of quantitative and qualitative data leading to the generation of quantitative and qualitative evidence, which are then integrated together). The decision as to which approach to use is dependent on the nature/type of questions that are posed in the systematic review. If the review question can be addressed by both quantitative and qualitative research designs, the convergent integrated approach should be followed; if the focus of the review is on different aspects or dimensions of a particular phenomenon of interest, the convergent segregated approach is undertaken. Some example review questions are provided below, which delineate the different approaches.

“What are the barriers and enablers to the adoption of electronic health records to support self-management in adult patients with a chronic disease?”

• Here the focus is on barriers and enablers, which can be addressed through qualitative research (e.g., through a phenomenological study of health care professionals involved in supporting adult patients with a chronic disease through the use of electronic health records) as well as quantitative research (e.g., through a survey of health care professionals involved in the use of electronic health records conducted as part of a cross sectional study).
• Since this review question can be answered by both quantitative AND qualitative studies, it would follow a convergent integrated approach to its synthesis and integration.

“What are the effects of canine-assisted interventions (CAIs) on the health and social care of older people residing in long-term care?” and “What is the experience of older people residing in long-term care who receive CAIs?”

• Here both questions relate to a common phenomenon (i.e., CAIs for older people), but they are addressing two different aspects associated with it: the effects these interventions have on older people (i.e., the effect of the interventions on outcomes such as stress and anxiety), and how older people experience or perceive them. Questions of effectiveness are answered through quantitative research (e.g., through a randomized controlled trial comparing CAIs with standard interventions), and questions of experience/perception are answered through qualitative research (e.g., through an ethnographic study where the researcher undertakes fieldwork on a group of older people receiving these interventions).
• Since this review focuses on different dimensions of a phenomenon, it would follow a convergent segregated approach to its synthesis and integration.

The methodological guidance for the synthesis and integration of these two approaches is presented separately in the succeeding sections.

The convergent integrated approach to synthesis and integration

The convergent integrated approach, outlined in example 1 above, refers to a process of combining extracted data from quantitative studies (including data from the quantitative component of mixed methods studies) and qualitative studies (including data from the qualitative component of mixed methods studies), and involves data transformation. In order for qualitative and quantitative data to be integrated and thus fully inform the topic, one approach is for the data to be transformed into a mutually compatible format. 27 Data transformation can occur either by converting qualitative data into quantitative data (i.e., quantitizing) or by converting quantitative data into qualitative data (i.e., qualitizing). Quantitizing is a process in which qualitative data are assigned numerical values, whereas qualitizing refers to quantitative data being converted into themes, categories, typologies, or narratives. 2,3,23

For data transformation, JBI recommends that quantitative data be qualitized, as codifying quantitative data is less error-prone than attributing numerical values to qualitative data. 22 Qualitizing involves extracting data from quantitative studies and translating or converting it into textual descriptions to allow integration with qualitative data. Qualitizing involves a narrative interpretation of the quantitative results. At the simplest level, qualitized data might comprise describing a sample (or members of it) using word categories based on supplementary descriptive statistics such as averages or percentage scores. 28 Qualitized data can also include profiling of the sample using cluster or factor analysis. 28 Data with a temporal or longitudinal component, 28 or those that examine associations and relationships using inferential statistics such as linear or logistic regression analysis, also have narrative potential and can therefore be qualitized by identifying variables included in the analysis. By qualitizing, the reviewer converts the quantities into declarative standalone sentences in a way that answers the review question.

The textual descriptions (qualitized data) from quantitative studies are then assembled and pooled with the qualitative data extracted directly from qualitative studies. Reviewers are then required to undertake repeated, detailed examination of the assembled data to identify categories on the basis of similarity in meaning, much like the process of meta-aggregation for qualitative synthesis. 29 A category will integrate two or more qualitative data, qualitized data or a combination of both. In some instances, however, data may not have the same meaning as others (i.e., may not reciprocally translate across studies) 30 and therefore cannot be combined to form a category. Where possible, categories are then aggregated to produce the overall integrated findings of the review. This process is illustrated in Figure 1 .

The convergent segregated approach to synthesis and integration

The convergent segregated approach consists of conducting separate quantitative synthesis and qualitative synthesis, followed by integration of evidence derived from both syntheses. By integrating the quantitative and qualitative synthesized findings, a greater depth of understanding of the phenomena of interest can be obtained, compared to undertaking two separate component syntheses without formally linking the two sets of evidence. The guidance developed for this approach currently focuses exclusively on reviews addressing questions of meaningfulness/experience (qualitative) and effectiveness (quantitative).

In example 2 above, quantitative data are synthesized in the form of a meta-analysis (or a narrative summary if meta-analysis is not possible) to determine the effects of canine-assisted interventions on older adults residing in long-term care. Additionally, the qualitative data are pooled (in the case of the JBI approach, through the process of meta-aggregation, or through a narrative summary if meta-aggregation is deemed inappropriate) to determine the experiences/perceptions of older adults receiving these interventions. There is no order to which synthesis is done first, as they are independent; however, both must be completed before moving to the next step: integration of quantitative and qualitative evidence. This next step involves juxtaposing the synthesized quantitative results with the synthesized qualitative findings, and organizing or linking the results and findings into a line or argument to produce an overall configured analysis. This is where the reviewer considers how (and if) the results and findings complement each other by using one type of evidence to explore, contextualize, or explain the findings of the other type of evidence. In this step, results and findings cannot be reduced but are organized into a coherent whole. 3 In this approach, the reviewer repeatedly compares the results of the quantitative synthesis with the findings of the qualitative synthesis, analyzing the intervention that had been investigated for effectiveness (quantitative) in light of the experiences of the participants (qualitative). The following questions act as a guide for this process:

• Are the results/findings from individual syntheses supportive or contradictory?
• Does the qualitative evidence explain why the intervention is or is not effective?
• Does the qualitative evidence help explain differences in the direction and size of effect across the included quantitative studies?
• Which aspects of the quantitative evidence are or are not explored in the qualitative studies?
• Which aspects of the qualitative evidence are or are not tested in the quantitative evidence?

In some instances, the reviewer may find that the results of the quantitative synthesis are not complementary or have no relationship with the findings of the qualitative synthesis, or vice versa. In such cases, the reviewer may identify gaps where further research may be useful to explain the contradictory findings or when there is no relationship between the qualitative findings and quantitative results. The JBI convergent segregated approach to synthesis and integration is illustrated in Figure 2 , while Figure 3 provides a summary of both approaches.

Mixed methods systematic reviews provide an innovative approach for addressing important questions in health care. 31 The increasing interest in this type of review, and the variability and lack of clear detail in the methods to synthesize quantitative and qualitative data or integrate quantitative and qualitative evidence indicate the need for guidance on how MMSRs should be undertaken. Based on a review of the international literature on MMSRs, and with input from experienced researchers in this field, JBI updated its methodological guidance and identified two synthesis designs for conducting MMSRs: convergent integrated, and convergent segregated.

The JBI methodological approach is based on the typology developed by Hong et al. 17 as well as the seminal work undertaken by Sandelowski and colleagues. 3,32 The convergent integrated approach is similar to Sandelowski's integrated design, which involves direct assimilation, and is based on the assumption that quantitative and qualitative data can both address the same research question. 3,32 As such, they can be combined once data have been transformed in the same format (i.e., quantitized or qualitized). Comparable to JBI's convergent integrated approach and Sandelowski's integrated design is the data-based convergent design identified by Hong et al. , 17 which typically involves a broad systematic review question (that can be answered by both quantitative and qualitative studies) and a synthesis that occurs following data extraction and transformation. 17 On the other hand, the convergent segregated approach is analogous to Sandelowski's segregated design. In contrast to the integrated design, which allows direct assimilation, the segregated design involves the integration of evidence through a method known as configuration. Configuration refers to the arrangement of complementary evidence into a line of argument. 3,32 According to Sandelowski et al. , complementarity is based on the assumption that quantitative and qualitative evidence address different research questions that are related to the same phenomenon of interest. 3,32 In other words, quantitative and qualitative evidence address different aspects or dimensions of a phenomenon of interest; therefore, they can neither corroborate nor refute each other but rather only complement each other. As such, the quantitative and qualitative evidence cannot be directly combined and can only be organized into a coherent whole. This approach to synthesis corresponds to Hong et al. 's 17 results-based convergent design that typically involves an overall systematic review question with sub-questions (some that can be addressed only by quantitative studies and others that can be addressed only by qualitative studies); there is a separate and simultaneous synthesis of quantitative and qualitative data, followed by the integration of the resulting quantitative and qualitative evidence.

Mixed methods systematic reviews appear to be the most complex and the least developed of all systematic review methods. The updated JBI guidance provides foundational work to this rapidly evolving methodology; however, it provides only a starting point for developing methods for combining quantitative and qualitative evidence in MMSRs, which may be conceived as a narrow conceptualization of mixed methods. It is hoped that in future iterations of JBI guidance, more sophisticated methods for integrating evidence will be developed and explored.

The methodological approach outlined in this paper also comes with some caveats. In the convergent segregated approach, the current JBI guidance specifically focuses on intervention/treatment or effectiveness questions for the quantitative component and on meaningfulness or experience questions for the qualitative component. However, the JBI Mixed Methods Review Methodology Group acknowledges that there are other types of review questions that lend themselves to a segregated approach. For example, an MMSR may ask a prevalence question or patterns of use of a specific treatment (which is quantitative in nature) along with the experiences of patients regarding that treatment (qualitative component). While the group believes that a segregated approach is broad enough to be applied to other types of MMSR questions, future iterations of the JBI methodology will provide explicit guidance on how such questions can be synthesized and integrated in MMSRs.

One of the distinguishing features of an MMSR is the inclusion of not only primary quantitative and qualitative studies but also primary mixed methods studies. For primary mixed methods studies included in a JBI MMSR, data are extracted so that they can be classified as quantitative or qualitative. In the integrated approach, quantitative data are then qualitized to allow synthesis, whereas in a segregated approach, data are kept separate and then go through either meta-analysis or meta-aggregation (as appropriate), followed by the integration of the resulting evidence. This approach of categorizing data into quantitative or qualitative, particularly for the segregated approach, is ideal for primary mixed methods studies in which the quantitative component is published separately from the qualitative component. This is usually the case for mixed methods research that applies a sequential explanatory design 33 (i.e., where qualitative findings are used to interpret or explain quantitative results). 34 However, for primary mixed methods research where the results represent the actual integration of the quantitative and qualitative data (such as those found in realist evaluation), categorizing data into quantitative or qualitative may not be ideal and philosophically would negate the strength of mixed methods studies. It would seem intuitive that in such instances, data are classified into three streams (i.e., quantitative, qualitative, and mixed methods), followed by a configurative analysis to allow integration. This will be future work for the JBI Mixed Methods Review Methodology Group.

In addition to those identified above, the JBI Mixed Methods Review Methodology Group has identified a number of methodological projects that need to be undertaken to advance this field. First, as with other systematic reviews, critical appraisal is an essential component of MMSRs, and currently JBI advocates the use of the appropriate JBI quantitative tools (for quantitative studies and the quantitative component of mixed methods studies) and the JBI qualitative tool (for qualitative studies and the qualitative component of mixed methods studies). It may be necessary to develop a bespoke tool for mixed methods primary studies or identify an already existing critical appraisal tool for use in JBI MMSRs. 24,25,35,36 Additionally, with regard to critical appraisal in the integrated approach, further investigation into how the appraisal results of quantitative studies (in which findings have been qualitized) are incorporated into the synthesis is needed.

One of the strengths of a systematic review, particularly JBI systematic reviews, is its ability to provide actionable and explicit practice recommendations. These recommendations are based on review findings that have been assessed using a structured approach: GRADE 37 for systematic reviews of effectiveness, and ConQual 38 for systematic reviews of qualitative studies. Due to the complexities associated with recommendations being derived from both streams of evidence and the impact of data transformation and/or integration on the grading process, an assessment of the certainty of the evidence using either the GRADE or ConQual approach is currently not recommended for JBI MMSRs following either the convergent integrated or convergent segregated approach. Modification to existing systems that assess the certainty of evidence may need to be investigated or, alternatively, a new system developed for evaluating results or findings from an MMSR. Finally, although this paper has focused on the conduct of reviews and not their reporting, it is evident that there is a lack of consensus in terms of reporting standards for MMSRs. This may be due to the lack of universally agreed and specific guidelines for such reviews. As the demand for this type of review increases along with significant methodological advancements in MMSRs, work can now be initiated to improve the standards for reporting.

This paper outlines an exciting development in the field of mixed methods synthesis. The update of the JBI methodological guidance for conducting an MMSR recommends that reviewers take a convergent approach to synthesis and integration whereby the specific method utilized is dictated by the nature/type of questions that are posed in the systematic review. If the review question can be addressed by both quantitative and qualitative research designs, then the convergent integrated approach should be followed, which involves data transformation and allows reviewers to combine quantitative and qualitative data. If the focus of the review is on different aspects or dimensions of a particular phenomenon of interest, then the convergent segregated approach should be undertaken, which involves independent synthesis of quantitative and qualitative data, leading to the generation of quantitative and qualitative evidence, which are then integrated together. Limitations to the current guidance are discussed, as are a series of methodological projects the Methodology Group will undertake to allow for further refinement of this methodology.

Acknowledgments

All past members of the JBI Mixed Methods Review Methodology Group, experts in the field who were contacted for advice, and those who provided feedback to members of the group when the guidance was presented at international scientific conferences.

• Cited Here |
• View Full Text | PubMed | CrossRef |
• Google Scholar
• PubMed | CrossRef |

∗ Lucylynn Lizarondo and Cindy Stern are co-first authors.

data transformation; integration; mixed methods; synthesis; systematic review

• + Favorites
• View in Gallery

Readers Of this Article Also Read

Qualitative research synthesis: methodological guidance for systematic..., the development of software to support multiple systematic review types: the..., the updated joanna briggs institute model of evidence-based healthcare, updated methodological guidance for the conduct of scoping reviews, the jbi model of evidence-based healthcare.

Systematic Reviews & Other Review Types

• What is a Systematic Review?
• What is a Scoping Review?
• What is a literature review?
• What is a Rapid Review?

What is a Mixed Methods Review?

• What is a Network Meta-Analysis?
• What is an Overview of Reviews?
• What is a Meta-Syntheses?
• What is an Integrative Review?
• What is a Diagnostic Test Accuracy Review?
• What is a Living Systematic Review?

A  comprehensive syntheses of two or more types of data (e.g. quantitative and qualitative) are conducted and then aggregated into a final, combined synthesis

Qualitative and quantitative data are combined and synthesized in a single primary synthesis.

(The Joanna Briggs Institute 2014 Reviewers Manual)

Mixed Methods Reviews are best designed for:

• Multidisciplinary topics or topics with a body of literature that includes quantitative, qualitative, and mixed methods studies.
• To determine not only the effects of interventions but also their appropriateness.
• To identify research gaps.
• To provide an explanation for possible heterogeneity between trials.
• To answer multiple questions in one systematic review.
• "Integrate quantitative estimates of benefit and harm with more qualitative understanding from people's lives."

(Harden A. 2010)

How a Mixed Methods Review differs from a Systematic Review

Timeframe:  12-18+ months.  Same as a systematic review or longer. 

Question:  Addresses 2 or more specific PICO criteria, "Mixed methods reviews should pose a question that specifically requires the inclusion of two or more syntheses that are grounded in different approaches".  A priori review protocol is recommended.

Examples of and clearly articulated PICO questions that may be posed by one mixed methods systematic review are:

1. What is the effectiveness of educational strategies associated with insulin pump therapy?

2. What is the appropriateness of educational strategies associated with insulin pump therapy”

Sources and searches:  Types of studies and their findings are mixed.  This requires a very broad search or multiple structured searches. "This should address each of the syntheses included in the review...[and] aims to find both published and unpublished studies."

Selection:  Based upon inclusion criteria.  

• "The quantitative component of the review will consider any experimental study design including randomized controlled trials, non-randomised controlled trials, quasi-experimental, before and after studies for inclusion."
• The qualitative component  The textual component of the review will consider expert opinion, discussion papers, position papers and other text.

Appraisal:  "This should address each of the syntheses included in the review."  The appropriate critical appraisal tool or method should be applied by study type or there should be one general tool used that will represent all study designs.  

Synthesis:  Depends on findings and included studies, applying mixed synthesis methods (qualitative:thematic synthesis/meta-ethnography & quantitative:statistical meta-analysis).  “The results of each single method synthesis included in the mixed method review will be extracted in numerical, tabular or textual format."

(Source: Joanna Briggs Institute 2014)

Mixed Methods Resources

• Mixed-Methods Systematic Reviews: Integrating Quantitative and Qualitative Findings
• Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute
• Methodology for JBI Mixed Methods Systematic Reviews

Limitations of a Mixed Methods Review

• May result in a larger number of citations.
• Requires more search time to create multiple searches for varying outcomes.
• There are no universally adopted methods for conducting Mixed Methods Reviews.
• Requires significant methodological skill.
• Resource intensive--may take time to engage with the evidence and develop theory.
• Not inherently reproducible or transparent because of the highly iterative nature of the interpretative process.

(Source: M. Petticrew et al (2013)

Other names for a Mixed Methods Review

Mixed Research Synthesis, Mixed Methods Synthesis, Mixed Methods Systematic Review, Mixed Studies Review, Qualitative & Quantitative Systematic Review

Temple Attribution

Adapted with permission from Temple University Libraries. https://guides.temple.edu/systematicreviews

• Last Updated: Mar 13, 2024 4:27 PM
• URL: https://touromed.libguides.com/review_types

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

A mixed-methods approach to systematic reviews

Affiliation.

• 1 1The Joanna Briggs Institute, University of Adelaide, Adelaide, South Australia, Australia 2The University of Nottingham Centre for Evidence Based Nursing and Midwifery: A JBI Collaborating Centre, University of Nottingham, Nottingham, England, UK 3The Northeast Institute for Evidence Synthesis and Translation (NEST): A JBI Collaborating Centre, Rutgers, The State University of New Jersey, New Jersey, USA 4Portugal Centre for Evidence Based Practice: A JBI Affiliate Centre, Escola Superior de Enfermagem de Coimbra, Coimbra, Portugal 5The Scottish Centre for Evidence-Based Multi-professional Practice: A JBI Collaborating Centre, Robert Gordon University, Aberdeen, Scotland, UK.
• PMID: 26196082
• DOI: 10.1097/XEB.0000000000000052

There are an increasing number of published single-method systematic reviews that focus on different types of evidence related to a particular topic. As policy makers and practitioners seek clear directions for decision-making from systematic reviews, it is likely that it will be increasingly difficult for them to identify 'what to do' if they are required to find and understand a plethora of syntheses related to a particular topic.Mixed-methods systematic reviews are designed to address this issue and have the potential to produce systematic reviews of direct relevance to policy makers and practitioners.On the basis of the recommendations of the Joanna Briggs Institute International Mixed Methods Reviews Methodology Group in 2012, the Institute adopted a segregated approach to mixed-methods synthesis as described by Sandelowski et al., which consists of separate syntheses of each component method of the review. Joanna Briggs Institute's mixed-methods synthesis of the findings of the separate syntheses uses a Bayesian approach to translate the findings of the initial quantitative synthesis into qualitative themes and pooling these with the findings of the initial qualitative synthesis.

PubMed Disclaimer

Similar articles

• Conducting systematic reviews of association (etiology): The Joanna Briggs Institute's approach. Moola S, Munn Z, Sears K, Sfetcu R, Currie M, Lisy K, Tufanaru C, Qureshi R, Mattis P, Mu P. Moola S, et al. Int J Evid Based Healthc. 2015 Sep;13(3):163-9. doi: 10.1097/XEB.0000000000000064. Int J Evid Based Healthc. 2015. PMID: 26262566
• Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Aromataris E, et al. Int J Evid Based Healthc. 2015 Sep;13(3):132-40. doi: 10.1097/XEB.0000000000000055. Int J Evid Based Healthc. 2015. PMID: 26360830
• Guidance for conducting systematic scoping reviews. Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Peters MD, et al. Int J Evid Based Healthc. 2015 Sep;13(3):141-6. doi: 10.1097/XEB.0000000000000050. Int J Evid Based Healthc. 2015. PMID: 26134548
• Synthesis of qualitative research and evidence-based nursing. Flemming K. Flemming K. Br J Nurs. 2007 May 24-Jun 13;16(10):616-20. doi: 10.12968/bjon.2007.16.10.23510. Br J Nurs. 2007. PMID: 17577168 Review.
• A scoping approach to systematically review published reviews: Adaptations and recommendations. Schultz A, Goertzen L, Rothney J, Wener P, Enns J, Halas G, Katz A. Schultz A, et al. Res Synth Methods. 2018 Mar;9(1):116-123. doi: 10.1002/jrsm.1272. Epub 2017 Nov 22. Res Synth Methods. 2018. PMID: 29032590 Review.
• Performance, impact and experiences of using wearable devices for seizure detection in community-based settings: a mixed methods systematic review. Sasseville M, Attisso E, Gagnon MP, Supper JM, Ouellet S, Amil S, Assi EB, Nguyen DK. Sasseville M, et al. Mhealth. 2024 Jul 12;10:27. doi: 10.21037/mhealth-24-7. eCollection 2024. Mhealth. 2024. PMID: 39114464 Free PMC article. Review.
• What is known about adolescent dysmenorrhoea in (and for) community health settings? Dixon S, Hirst J, Taghinejadi N, Duddy C, Vincent K, Ziebland S. Dixon S, et al. Front Reprod Health. 2024 Jul 23;6:1394978. doi: 10.3389/frph.2024.1394978. eCollection 2024. Front Reprod Health. 2024. PMID: 39109074 Free PMC article.
• Help-seeking for cognitive problems in older adults without dementia: A systematic review. Hill NL, Bratlee-Whitaker E, Sillner A, Brautigam L, Mogle J. Hill NL, et al. Int J Nurs Stud Adv. 2021 Nov 3;3:100050. doi: 10.1016/j.ijnsa.2021.100050. eCollection 2021 Nov. Int J Nurs Stud Adv. 2021. PMID: 38746728 Free PMC article. Review.
• Effect of interventions to improve safety culture on healthcare workers in hospital settings: a systematic review of the international literature. Finn M, Walsh A, Rafter N, Mellon L, Chong HY, Naji A, O'Brien N, Williams DJ, McCarthy SE. Finn M, et al. BMJ Open Qual. 2024 May 7;13(2):e002506. doi: 10.1136/bmjoq-2023-002506. BMJ Open Qual. 2024. PMID: 38719514 Free PMC article.
• Knowledge, perceptions and practices towards diabetes risk in sub-Saharan Africa: a mixed-methods scoping review. Manyara AM, Mwaniki E, Gill JM, Gray CM. Manyara AM, et al. Public Health Nutr. 2024 Mar 27;27(1):e104. doi: 10.1017/S1368980024000752. Public Health Nutr. 2024. PMID: 38533768 Free PMC article. Review.
• Search in MeSH

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.
• Wolters Kluwer

Research Materials

• NCI CPTC Antibody Characterization Program

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals

You are here

• Volume 20, Issue 3
• Mixed methods research: expanding the evidence base
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Allison Shorten 1 ,
• Joanna Smith 2
• 1 School of Nursing , University of Alabama at Birmingham , USA
• 2 Children's Nursing, School of Healthcare , University of Leeds , UK
• Correspondence to Dr Allison Shorten, School of Nursing, University of Alabama at Birmingham, 1720 2nd Ave South, Birmingham, AL, 35294, USA; [email protected]; ashorten{at}uab.edu

https://doi.org/10.1136/eb-2017-102699

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

‘Mixed methods’ is a research approach whereby researchers collect and analyse both quantitative and qualitative data within the same study. 1 2 Growth of mixed methods research in nursing and healthcare has occurred at a time of internationally increasing complexity in healthcare delivery. Mixed methods research draws on potential strengths of both qualitative and quantitative methods, 3 allowing researchers to explore diverse perspectives and uncover relationships that exist between the intricate layers of our multifaceted research questions. As providers and policy makers strive to ensure quality and safety for patients and families, researchers can use mixed methods to explore contemporary healthcare trends and practices across increasingly diverse practice settings.

What is mixed methods research?

Mixed methods research requires a purposeful mixing of methods in data collection, data analysis and interpretation of the evidence. The key word is ‘mixed’, as an essential step in the mixed methods approach is data linkage, or integration at an appropriate stage in the research process. 4 Purposeful data integration enables researchers to seek a more panoramic view of their research landscape, viewing phenomena from different viewpoints and through diverse research lenses. For example, in a randomised controlled trial (RCT) evaluating a decision aid for women making choices about birth after caesarean, quantitative data were collected to assess knowledge change, levels of decisional conflict, birth choices and outcomes. 5 Qualitative narrative data were collected to gain insight into women’s decision-making experiences and factors that influenced their choices for mode of birth. 5

In contrast, multimethod research uses a single research paradigm, either quantitative or qualitative. Data are collected and analysed using different methods within the same paradigm. 6 7 For example, in a multimethods qualitative study investigating parent–professional shared decision-making regarding diagnosis of suspected shunt malfunction in children, data collection included audio recordings of admission consultations and interviews 1 week post consultation, with interactions analysed using conversational analysis and the framework approach for the interview data. 8

What are the strengths and challenges in using mixed methods?

Selecting the right research method starts with identifying the research question and study aims. A mixed methods design is appropriate for answering research questions that neither quantitative nor qualitative methods could answer alone. 4 9–11 Mixed methods can be used to gain a better understanding of connections or contradictions between qualitative and quantitative data; they can provide opportunities for participants to have a strong voice and share their experiences across the research process, and they can facilitate different avenues of exploration that enrich the evidence and enable questions to be answered more deeply. 11 Mixed methods can facilitate greater scholarly interaction and enrich the experiences of researchers as different perspectives illuminate the issues being studied. 11

The process of mixing methods within one study, however, can add to the complexity of conducting research. It often requires more resources (time and personnel) and additional research training, as multidisciplinary research teams need to become conversant with alternative research paradigms and different approaches to sample selection, data collection, data analysis and data synthesis or integration. 11

What are the different types of mixed methods designs?

Mixed methods research comprises different types of design categories, including explanatory, exploratory, parallel and nested (embedded) designs. 2   Table 1 summarises the characteristics of each design, the process used and models of connecting or integrating data. For each type of research, an example was created to illustrate how each study design might be applied to address similar but different nursing research aims within the same general nursing research area.

• View inline

Types of mixed methods designs*

What should be considered when evaluating mixed methods research?

When reading mixed methods research or writing a proposal using mixed methods to answer a research question, the six questions below are a useful guide 12 :

Does the research question justify the use of mixed methods?

Is the method sequence clearly described, logical in flow and well aligned with study aims?

Is data collection and analysis clearly described and well aligned with study aims?

Does one method dominate the other or are they equally important?

Did the use of one method limit or confound the other method?

When, how and by whom is data integration (mixing) achieved?

For more detail of the evaluation guide, refer to the McMaster University Mixed Methods Appraisal Tool. 12 The quality checklist for appraising published mixed methods research could also be used as a design checklist when planning mixed methods studies.

• Elliot AE , et al
• Creswell JW ,
• Plano ClarkV L
• Greene JC ,
• Caracelli VJ ,
• Ivankova NV
• Shorten A ,
• Shorten B ,
• Halcomb E ,
• Cheater F ,
• Bekker H , et al
• Tashakkori A ,
• Creswell JW
• 12. ↵ National Collaborating Centre for Methods and Tools . Appraising qualitative, quantitative, and mixed methods studies included in mixed studies reviews: the MMAT . Hamilton, ON : BMJ Publishing Group , 2015 . http://www.nccmt.ca/resources/search/232 (accessed May 2017) .

Competing interests None declared.

Provenance and peer review Commissioned; internally peer reviewed.

Read the full text or download the PDF:

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Mixed Methods Research | Definition, Guide & Examples

Mixed Methods Research | Definition, Guide & Examples

Published on August 13, 2021 by Tegan George . Revised on June 22, 2023.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, advantages of mixed methods research, disadvantages of mixed methods research, other interesting articles, frequently asked questions.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalizability : Qualitative research usually has a smaller sample size , and thus is not generalizable. In mixed methods research, this comparative weakness is mitigated by the comparative strength of “large N,” externally valid quantitative research.
• Contextualization: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help “put meat on the bones” of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

But mixed methods might be a good choice if you want to meaningfully integrate both of these questions in one research study.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions.

Mixed methods can be very challenging to put into practice, and comes with the same risk of research biases as standalone studies, so it’s a less common choice than standalone qualitative or qualitative research.

Prevent plagiarism. Run a free check.

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyze them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyze cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyze accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyze both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualize your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses . Then you can use the quantitative data to test or confirm your qualitative findings.

“Best of both worlds” analysis

Combining the two types of data means you benefit from both the detailed, contextualized insights of qualitative data and the generalizable , externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalizable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labor-intensive. Collecting, analyzing, and synthesizing two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Due to the fact that quantitative and qualitative data take two vastly different forms, it can also be difficult to find ways to systematically compare the results, putting your data at risk for bias in the interpretation stage.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analyzed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analyzed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualize your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analyzed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). Mixed Methods Research | Definition, Guide & Examples. Scribbr. Retrieved September 6, 2024, from https://www.scribbr.com/methodology/mixed-methods-research/

Is this article helpful?

Tegan George

Other students also liked, writing strong research questions | criteria & examples, what is quantitative research | definition, uses & methods, what is qualitative research | methods & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Acta Inform Med
• v.30(1); 2022 Mar

Mixed Methodology of Scientific Research in Healthcare

Emina smajic.

1 Agram Polyclinic, Sarajevo, Bosnia and Herzegovina

Dijana Avdic

2 Faculty of Health Studies, Sarajevo, Bosnia and Herzegovina

Aleksandra Pasic

3 Clinical Biochemistry with Immunology, Sarajevo, Clinical Center University of Sarajevo, Bosnia and Herzegovina

Alden Prcic

4 General Hospital “Prim dr. Abdulah Nakas”, Sarajevo, Bosnia and Herzegovina

Maja Stancic

Background:.

Scientific research is usually classified as quantitative or qualitative. However, methodologists are increasingly emphasizing the integration of qualitative and quantitative data as the center of mixed methods (mix methodologies). Mixed research method implies the use of different research methods, ie. quantitative and qualitative methods in one study.

The aim of this review paper is to present the purpose of using a mixed methodology in health research.

The relevant articles were searched from online data sources including PubMed and Google Scholar.

This approach to the use of mixed methods creates opportunities for a deeper study of various problems. The purpose of using mixed research methods is to obtain valid answers to research questions, however the researcher may still have different reasons or purposes for which he wants to strengthen the research study and its conclusions by applying mixed methods. The use of mixed scientific methodology is widely used in the field of health outcomes and should not be limited to a closed list of possible methodological options.

Conclusion:

Recently, there has been an increase in the number of scientific studies in healthcare that use mixed research methods. The advantage of applying this scientific method is that through the triangulation of data obtained by different (quantitative / qualitative) approaches, we get a deeper and more complete picture of the phenomenon in health care that we observe.

1. BACKGROUND

In health science research, there is a priority to develop new methodologies to improve the quality and scientific strength of data leading to an extraordinary increase in methodological diversity. This diversity reflects the nature of public health problems, such as differences between populations, age groups, ethnic groups and cultures, poor adherence to treatments considered effective, behavioral factors contributing to disability and health, and translational needs for health research. Diversity also signals a growing acceptance of qualitative and social science research, the formation of interdisciplinary research teams, and the use of multilevel approaches to research complex health issues such as patient attitudes and cultural and social models of disease and health (1) .

2. OBJECTIVE

The retrieved articles were reviewed by the authors and the results are presented along with the relevant discussion

4.1. MIXED METHODOLOGY

Scientific research is usually classified as quantitative or qualitative. However, methodologists are increasingly emphasizing the integration of qualitative and quantitative data as the center of mixed methods (mix methodologies). Integration is a deliberate process by which the researcher combines quantitative and qualitative approaches in the study. Quantitative and qualitative data then become interdependent in solving questions and hypotheses. Mixed research method implies the use of different research methods, quantitative and qualitative methods in one study ( 2 , 3 ). Research on mixed methods should be distinguished from multi-method research (method-combination) in which either multiple qualitative approaches or only multiple quantitative approaches are combined (4) .

The most accepted definition of a mixed research method is that it is a research in which a researcher or team of researchers combines elements of a qualitative and quantitative approach to research (use of qualitative and quantitative perspectives, data collection, analysis, inference techniques) to understand and support research. As we see in the definition, the use of both quantitative and qualitative methods in a single study (or series of related studies) is crucial, unlike the use of combined methods that combine two or more quantitative or two or more qualitative research methods ( 2 , 4 ).

The basic premise of using mixed research methods is that some research issues can be addressed more comprehensively than using either quantitative or qualitative methods. The issues that benefit most from the design of mixed methods tend to be broad and complex, with multiple aspects that each can have. Mixed research methods can exploit the strengths and weaknesses of both approaches and can be particularly useful when addressing complex, multifaceted issues such as health service interventions and living with chronic diseases (2) . This approach to the use of mixed methods creates opportunities for a deeper study of various problems (5) .

4.2. PURPOSE OF USING MIXED RESEARCH

The purpose of using mixed research methods is to obtain valid answers to research questions, however the researcher may still have different reasons or purposes for which he wants to strengthen the research study and its conclusions by applying mixed methods. The purpose classification of mixed research methods was first introduced in 1989 by Greene, Caracelli, and Graham, based on an analysis of published studies of mixed methods. This classification is still used and we have a total of five “purposes” for why a mixed methodology is used in research (4) . Classification of the purpose of using mixed methodology:

• Complementarity. Using data obtained from one method to illustrate the results of another method.
• Development. Using the results of one method to develop or inform about the use of another method.
• Initiation. We can use the results of different methods to search for areas of non-compliance in certain areas in order to create new insights.
• Expansion. Examining different aspects of a research question, with each aspect justifying different methods.
• Triangulation. Use of data obtained by both methods to support the findings (2) .

In the last 28 years, this classification has been supplemented by several other authors. So in 2006 Bryman compiled a list of more specific rationales for the use or purpose of mixed research methods. Bryman’s classification decomposes the categorization of Greene et al. on several aspects and adds a number of additional aspects (3) .

Bryman’s addition to the classification of Greene et al.

Credibility. It refers to suggestions that the application of both methodological approaches improves the integrity of the results.

Context. Refers to cases where the combination is justified in terms of qualitative research that provides contextual understanding, along with generalized, externally valid results, or broad relationships between the variables identified by the survey.

Illustration. It refers to the use of qualitative data to illustrate quantitative findings, often referred to as putting “meat on the bone” “dry” quantitative findings.

Usefulness or improvement of usefulness of results. It refers to the suggestion, which is more likely to be highlighted among articles with an applied focus, that combining the two approaches will be more useful for practitioners.

Confirmation and discovery. It involves the use of qualitative data to generate hypotheses and the use of quantitative research to test them within a single project.

Diversity of views. This includes two slightly different explanations–namely, combining the perspective of researchers and participants through quantitative and qualitative research, and discovering the relationship between variables through quantitative research, while revealing meanings among research participants through qualitative research (3) .

4.3. RESEARCH DESIGN

Research designs are procedures for collecting, analyzing, interpreting, and reporting data in research studies. They represent different models for doing research, and these models have distinct names and procedures associated with them (6) .

The four major types of mixed methods designs are the Triangulation Design, the Embedded Design, the Explanatory Design, and the Exploratory Design (6) .

4.4. HEALTH APPLICATION

There is a wide range of methods used to collect both quantitative and qualitative data. And the research question and the necessary data are the main determinants of the methods used. To a lesser extent, the choice of methods may be influenced by feasibility. Method priority refers to the emphasis on each method in the study. For example, a study may be predominantly quantitative with a small qualitative component or vice versa. Alternatively, both quantitative and qualitative methods and data may be equally weighted. The emphasis on each methodological component of the study will be driven mainly by the research question, research team skills and feasibility. Finally, researchers must decide when each method will be used in research (2) . By analyzing the research methods and research designs used, Bryman suggests that on the quantitative side, structured interview and questionnaire research within cross-sectional design predominates, while on the qualitative side, semi-structured interviews within cross-sectional design predominate (7) . A key feature of mixed-method research is its methodological pluralism, which often results in research that provides broader perspectives than those offered by monomethodal designs (8) .

The use of mixed scientific methodology is widely used in the field of health outcomes and should not be limited to a closed list of possible methodological options, but should be seen as a framework for a specific research issue to be addressed using quantitative and qualitative components (data and / or methods) , when quantitative and qualitative components are articulated intentionally and prospectively in a well-defined, pre-specified research design and as a framework for meta-inference (9) .

The importance of applying mixed research methods can be reflected in the trend of “measuring / analysing what is important” for patients and the treatment outcomes reported by patients are increasingly used in clinical care and research. However, a recent review of studies documenting the development of outcome measures reported by patients highlights that only 11% of them were developed actually asking patients which outcomes are important to them. This emphasizes the importance of applying mixed qualitative and quantitative methods in health research to ensure a focus on the priorities identified by the patient, scientific rigor, and improved patient outcomes (9) . Mixed methods are also an increasingly accepted approach used to investigate organizational phenomena in health care (10) .

The application of the mix methodology is considered a significant contribution to health science. By combining quantitative and qualitative data in the same study, health researchers can reap the benefits of each approach while minimizing their disadvantages. In practice, this endeavor facilitates research by health science researchers on the complex and multifactorial nature of human health and disease. Researchers using the mixed method approach for the first time can easily feel overwhelmed by uncertainty about the philosophical foundations of the method, as well as the multitude of typologies of mixed method research. Although further research and debate are warranted, health researchers seeking solutions to real problems are increasingly embracing pragmatism as a paradigm of choice (11) .

Author’s contribution:

All authors were involved in preparation of this article. Final proofreading was made by first author.

Financial support and sponsorship:

Conflicts of interest:.

There are no conflicts of interest.

• Systematic Review
• Open access
• Published: 04 September 2024

Experiences of menopausal transition among populations exposed to chronic psychosocial stress in the United States: a scoping review

• Emma A. Blackson 1 , 2   na1 ,
• Christina McCarthy 3   na1 ,
• Caryn Bell 1 ,
• Shokufeh Ramirez 1 , 2 &
• Alessandra N. Bazzano 1 , 2  

BMC Women's Health volume  24 , Article number:  487 ( 2024 ) Cite this article

Metrics details

The transition to menopause is a significant event affecting health, well-being, and quality of life. Menopause typically occurs between the ages of 44–57, accompanied by symptoms such as hot flashes, mood changes, and sleep disturbances. Being postmenopausal also increases the risk of cardiovascular disease, stroke, and osteoporosis. Despite its importance, menopause is under-researched and under-discussed, particularly concerning the impact of chronic psychosocial stress.

A scoping review of qualitative, quantitative, and mixed methods research was conducted to map existing literature on the transition to menopause among populations experiencing chronic psychosocial stress in the United States. The review followed the PRISMA-ScR methodology, systematically searching literature in PubMed and SCOPUS databases using MeSH terms. Studies were included which focused on menopausal symptoms and psychosocial stressors. Data extraction and charting were performed using Covidence software.

Fifteen studies were included, highlighting relationships between socioeconomic status, intimate partner violence, childhood abuse, and racial disparities which influenced menopausal experiences. Lower- income, higher perceived stress, and negative attitudes towards menopause were associated with increased psychological and somatic symptoms and early onset of menopause (prior to age 45). African American women were found to experience earlier onset and more severe vasomotor symptoms compared to their White counterparts. Women veterans used hormone therapy more frequently than the general population, particularly those with mood or anxiety disorders. The review also identified a geographic bias, with most studies conducted in the Northeast, Midwest, and Western regions of the United States.

Conclusions

This review underscores the necessity of considering social, cultural, and environmental factors in understanding menopausal experiences and addressing health disparities. Future research should aim to include diverse populations and adopt longitudinal and qualitative study designs to capture the dynamic nature of menopausal experiences. Policies and interventions directed at improving the well-being of women experiencing menopause in the context of chronic psychosocial stress are warranted.

Trial Registration

Peer Review reports

Introduction

The transition to menopause is a significant life alteration for all who experience it and it impacts health, wellbeing, and quality of life. Menopause is a biological stage that marks the cessation of menstruation for at least 12 consecutive months, typically occurring between ages 44–57 [ 1 ]. Common symptoms include hot flashes, night sweats, mood changes, and sleep disturbances. Additionally, some women may experience vaginal dryness, weight gain, and thinning hair, all of which are primarily driven by the decline in estrogen levels during this transitional phase [ 2 ]. Importantly, the transition to this stage may occur over a decade or more. A key life course milestone, the onset of menopause can increase the risk for various health issues such as cardiovascular disease, stroke, and osteoporosis [ 3 , 4 , 5 ]. Compared with other life stages, menopause receives less attention in medical training curricula and is an under-discussed topic in media and patient education [ 6 , 7 ]. A key barrier to improving understanding is that research on menopause has been historically underfunded compared with other health topics, leading to significant gaps in our knowledge of this important life stage [ 8 ]. The lack of attention to the transition is concerning as it can potentially limit quality of life and appropriate healthcare for women during this critical phase. As with other reproductive health issues, significant disparities are observed concerning menopause, thus it is an important topic of public health research and intervention [ 9 ].

Little is available in the scientific literature about the influence of societal factors on the transition to menopause and the lived experiences of people going through perimenopause/menopause. This transition is not just a biological process but is also affected by various complex social and psychological contexts [ 10 ]. In recent decades, greater recognition of the importance of social, emotional, and physical changes of midlife to a woman’s overall health and well-being has developed [ 11 ]. Studies have shown that demographic characteristics like socioeconomic status as well as health-related factors can influence how a woman experiences menopause, including the timing of natural menopause [ 12 , 13 , 14 ]. Recent studies also have begun addressing the timing of natural menopause and how it may differ among varying populations of women. Henderson et al. [ 15 ]. reported race/ethnicity as a significant independent predictor of the timing of natural menopause, supporting the hypothesis that the timing of natural menopause is driven by a combination of biological, reproductive, and lifestyle factors. However, most studies in the current literature focus on identifying the clinical symptomatology and biological determinants associated with the transition to menopause, with less attention paid to psychosocial factors. Studies have shown that chronic psychosocial stress is associated with characteristics of menopause [ 16 , 17 ]. For example, studies show that experiencing intimate partner violence is associated with how menopause is experienced [ 12 ]. Disparities in menopause experiences by social identities, like race/ethnicity and socioeconomic status, may also be implicated in experiences of chronic psychosocial stress [ 9 ].

Two underlying theoretical paradigms are critical to understanding the role of chronic psychosocial stress and inequities in menopause: intersectionality and life course theory. Intersectionality is a theoretical lens related to socially rooted mechanisms of inequality that are understood to be experienced at multiple nexuses of gender, race, and class. Thus these ‘intersections’ result in more than the sum of sexism and racism, for example, disability, housing status, sexuality, and other subjectivities are also included in women’s social experiences and sources of oppression. In the 1970s and 1980s, intersectionality was described by women of color scholars including Collins, [ 18 ] Moraga, [ 19 ]and Crenshaw [ 20 ] and has become an important lens for addressing health experiences. Identities, oppressions, and privileges affect each other and resulting inequities are cumulative affecting lived experiences, health and wellbeing.

Life course theory as drawn from the reproductive and maternal child health literature of the last two decades [ 21 , 22 , 23 ], is distinct from sociological life course theory as described by Elder [ 24 ] but related. Life Course Theory (LCT) examines health as an integrated continuum, as opposed to a series of distinct and independent stages. It posits that there is a dynamic interaction of social and environmental elements with biological, behavioral, and psychological factors that shape health outcomes over the course of an individual’s lifetime [ 25 ]. According to this view, factors exert effect on a trajectory of health throughout the life course. The menopausal transition is a period in midlife that shapes the trajectory of a woman’s life course. By considering both Intersectionality and Life Course Theory, a more comprehensive understanding of how psychosocial stress and inequities play a major role in this important phase of a woman’s life can be gained.

The purpose of this review is to map the existing literature on how the transition to menopause is experienced by individuals in the United States who are exposed to chronic psychosocial stress. The goal is to gain a better understanding of the relationship between such stress and the transition to menopause, in order to identify areas where gaps in the existing knowledge base, and suggest future research directions that can improve healthcare for women.

In order to map the existing literature on transition to menopause among populations experiencing chronic psychosocial stress, a scoping review of qualitative, quantitative, and mixed methods research was conducted. The term women in this article refers to persons assigned female at birth and/or who experience cessation of menstruation regardless of current gender identity. This review followed the methodology described in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews (PRISMA-ScR) and adapted the framework outlined by Arksey and O’Malley [ 26 ], along with the process described by Peters et al [ 27 ] and Tricco et al. [ 28 ]. The scoping review process including the following steps: (1) identification of the research; (2) identification of the relevant studies; (3) selection of studies; (4) charting of the data; and (5) collating, summarizing, and reporting the results [ 27 ].

Determining the research question

The following research questions were formulated to guide the scoping review:

What evidence exists describing experiences of transition to menopause (TM) among populations exposed to chronic psychosocial stress in the United States?

What are the proposed explanations for the association between psychosocial stress and menopause experiences in the literature, and what future research priorities does this suggest?

Identifying relevant literature

The authors worked with a public health librarian to outline the search parameters and strategy. The following databases were searched: PubMed and Scopus, with an initial search using the National Library of Medicine’s Medical Subjects Headings (MeSH) terms to uncover terms that were associated with research questions were conducted at the start of the summer of 2022. Literature published from 1992 until present was considered. Once the terms were selected, the literature was systematically searched in the PubMed and SCOPUS databases.

Searches were conducted using the following key words or phrases, ((“Women“[Mesh]) AND ( “Menopause“[Mesh] OR “Menopause, Premature“[Mesh] OR “Postmenopause“[Mesh] OR “Premenopause“[Mesh] )) AND (((((“Blacks“[Mesh]) OR “Hispanic or Latino“[Majr]) OR “Homeless Persons“[Mesh]) OR “Veterans“[Majr]) OR ( “Stress Disorders, Traumatic, Acute“[Mesh] OR “Financial Stress“[Mesh] OR “Stress Disorders, Traumatic“[Mesh] OR “Stress, Psychological“[Mesh] OR “Occupational Stress“[Mesh] OR “Stress Disorders, Post-Traumatic“[Mesh] )) OR ((“Racism“[Mesh] OR “Systemic Racism“[Mesh]) OR ( “Discrimination, Psychological“[Mesh] OR “Social Discrimination“[Mesh] OR “Perceived Discrimination“[Mesh] OR “Ageism“[Mesh] ))).

Selecting the studies

Studies were included if: (1) the population or sample included women or people who experience menopause, (2) the study included information on earlier onset, longer duration, or subjectively different or worse clinical outcomes of perimenopause, menopause, or postmenopause, and (3) included context around chronic psychosocial stressors such as violence, trauma, anti-Black or Latine racism or discrimination, low socioeconomic status, veterans’ status, or housing status. Studies were excluded if: (1) the article was not in English, (2) the participants were not in the United States, or (3) the article did not explicitly mention menopausal symptoms.

Including premenopausal women allowed for representation of early menopausal symptoms influenced by chronic psychosocial stress, providing a comprehensive view of the menopausal transition. This broader perspective informs how chronic stress impacts the onset and experience of menopause, enhancing future research and healthcare practices.

Data extraction

Data extraction and analysis were completed utilizing Covidence software. The citations from both PubMed and results were downloaded to Zotero Reference Software and exported to Covidence. Covidence is a web-based collaboration software platform that streamlines the production of systematic and other literature reviews. 28 For this review, after the articles were uploaded, duplicates were removed, and abstracts were screened by two principal reviewers for eligibility. A third reviewer resolved any conflicts regarding eligibility for inclusion. There were sixteen articles left for extraction of details, including study characteristics, participant demographics, and key findings.

Collate, summarize, and report the results

The final search results are displayed in the PRISMA-ScR Flow diagram (Fig.  1 ).

The final search results are displayed in the PRISMA-ScR Flow diagram

Geographic distribution

Of the fifteen studies included in this scoping review, six were conducted in national or multi-city samples; four were conducted in the western region of the United States (U.S.) which includes California, Washington, and Arizona; one study took place in the Midwestern part of the US, specifically in Ohio; and two were in Northeastern region of the US, specifically in Pittsburgh.

Study design and population

Of the fifteen studies, eight studies were cohort studies, six were cross-sectional studies, and one was a qualitative study. Within the studies identified for this scoping review, the participants were women of various races and ethnicities between the ages of 35–65. Studies included participants who were premenopausal, transitioning to menopause, or menopausal.

The studies looked at different aspects of women’s experiences during menopause, including their backgrounds, such as race, ethnicity, and socioeconomic status. Some studies focused on factors influencing menopausal symptoms, such as smoking, reproductive history, childhood abuse, vasomotor symptoms, and psychological distress. Other studies looked at broader aspects of midlife, such as predicting the menopausal transition, intimate partner violence, PTSD, and changes in sexual function. One investigation studied hormone replacement therapy among female veterans and how it affected their mental health. Additionally, studies explored differences in menopausal symptoms, self-care, and perceptions across racial and socioeconomic lines, providing insight into how women coped. Table  1 below contains details about the studies included in this scoping review.

Table  2 offers a comprehensive overview of studies that examined the relationship between psychosocial stressors and menopausal symptoms from the review. Each entry in the table includes the study authors and title, identifies the specific psychosocial stressors studied, and outlines the menopausal symptoms associated with these stressors. The table includes a variety of stressors, including socioeconomic status, racial discrimination, psychological distress, intimate partner violence, and childhood abuse, among others.

Study domains

Studies highlighted the association between socioeconomic factors and the experience of menopausal symptoms. DeMello [ 13 ] found that in Arizona, women from lower socioeconomic backgrounds, specifically those uninsured or homeless, reported heightened psychological and somatic menopausal symptoms, regardless of age, race, menopausal stage, or hormone therapy use. Similarly, Nosek [ 29 ] identified that a woman’s socioeconomic status, combined with her perceived stress and attitudes toward aging and menopause, played a significant role in shaping her menopausal symptom experience.

Among the studies reviewed, Gerber [ 30 ] conducted research that specifically examined the impact of hormone therapy on menopausal symptoms in women veterans. Their research revealed that hormone therapy was utilized by a significantly higher number of women veterans compared to the general population. Furthermore, their findings showed that women veterans who were diagnosed with mood or anxiety disorders were more likely to opt for hormone therapy. This correlation remained even after accounting for other factors, such as demographics and medical history.

Studies that focused on the effects of intimate partner violence and abuse on the menopausal transition found that a lifetime history of intimate partner violence or sexual assault was associated with increased severity of menopausal symptoms [ 12 , 31 ].

Four of the fifteen studies found psychological distress and a decline in mental health to be correlated with increased severity of menopausal symptoms including hot flashes, night sweats, depressed mood, and irregular menses during midlife [ 30 , 31 , 32 , 33 ].

Studies that addressed racial and ethnic differences reported that African American women were more likely to reach menopause earlier and experience more vasomotor symptoms than White women [ 34 , 35 , 36 , 37 ].

Some of the included studies highlighted changes in sexual function during menopause, [ 38 ] self-care management strategies during late pre-menopause and early peri-menopause, [ 37 ] and the knowledge, attitudes, and practices of African American women toward menopausal health [ 39 ].

This scoping review sought to examine the literature available on transition to menopause among populations experiencing chronic psychosocial stress in the US. Several findings are notable. First, socioeconomic position emerged as a significant factor influencing menopausal experiences, with lower incomes, higher perceived stress, and negative attitudes towards menopause associated with increased psychological and somatic symptoms and early-onset menopause [ 13 , 36 , 38 , 40 ]. This aligns with findings from 2023 by Michopoulos et al. [ 41 ], which demonstrated that perimenopausal women, particularly those from urban, low socioeconomic backgrounds, experienced significantly higher PTSD and depression symptoms compared to premenopausal and postmenopausal women. This suggests that chronic stress and trauma, prevalent in lower socioeconomic populations, exacerbate menopausal symptoms. Next, associations were also found between intimate partner violence and abuse and the severity of menopausal symptoms, namely that experiences of childhood abuse and neglect, or history of intimate partner violence or sexual assault were associated with increased menopausal symptoms [ 12 , 34 ]. One study reviewed the usage prevalence of hormone therapy among female Veterans seeking care from the Veterans Administration exceeded that of the general population by more than two-fold, implying that Veterans experience high levels of stress and thus are more likely to experience menopausal systems, requiring hormonal therapy [ 40 ]. Lastly, several studies highlighted racial and ethnic differences in experiences with the transition to menopause [ 31 , 32 , 35 , 37 , 42 ]. For instance, Black women, on average experienced earlier onset and more vasomotor symptoms, hot flashes and night sweats, than White women [ 36 ]. Williams et al. (2022) [ 43 ] further emphasize that African American women often face poorer health outcomes during menopause due to chronic exposure to racial stress, as suggested by the ‘weathering hypothesis’ [add ref here to Geronimus] This chronic stress accelerates aging and increases the risk of chronic conditions, contributing to more severe menopausal symptoms compared to their White counterparts. These findings emphasize the necessity of incorporating the intersectionality of social identities and sources of racialized and other forms of psychosocial stress while assessing menopausal experiences among populations. Lewis-Johnson et al. (2023) [ 44 ] found that everyday discrimination significantly impacts mental health during the menopausal transition, particularly among racially minoritized women. This supports the need to consider racial discrimination as a critical factor influencing menopausal experiences and outcomes.

The findings mentioned above are closely related to the theoretical frameworks discussed in the introduction of this scoping review, namely intersectionality and life course theory. Intersectionality highlights the complex interplay of various identities, such as gender, race, and class, that influence menopausal experiences [ 39 , 45 ]. The factors identified in this study, including socioeconomic status, intimate partner violence, and racial disparities, demonstrate the cumulative impact of multiple forms of oppression and marginalization. Similarly, life course theory emphasizes the dynamic interaction of social, environmental, and individual factors over time [ 46 , 47 ]. This theory underscores the importance of viewing menopausal experiences as part of a continuum and recognizing the impact of early-life experiences and social determinants of health in shaping health outcomes and how the transition is experienced during the menopausal transition.

Beyond the findings and theoretical underpinnings, this scoping review highlights a geographic skew in the included studies. They predominantly include participants from the Northeast, Midwest, and Western regions of the United States, leaving gaps in understanding of menopausal experiences among those in the Southern region [ 12 , 13 , 30 , 32 , 33 , 34 , 36 , 38 , 42 , 48 ]. Additionally, studies like those reviewed by Williams et al. (2022) [ 43 ] indicate that women in the Southern United States, particularly African American women, experience earlier menopause onset. The South, and particularly the Gulf South, a region of the United States with a large low-income and racially minoritized population has high rates of poor health outcomes for women, including reproductive health outcomes as well as cardiovascular disease outcomes [ 3 , 4 , 5 ]. Future research must strive to bridge this gap, ensuring a broader representation of the menopausal experience among populations exposed to psychosocial stress across various geographical landscapes.

There is a need to examine the complexity of menopausal transitions from multiple perspectives, through rigorous study designs and including diverse population characteristics. Additional study designs are needed to further elucidate the unique nature of menopausal transition in the context of psychosocial stress, particularly longitudinal studies and qualitative studies are warranted. Longitudinal, or cohort studies, can track the progression of symptoms and stressors over time, while qualitative studies offer a personalized understanding of women’s experiences [ 49 , 50 ]. Longitudinal studies are crucial for tracking changes over time. It’s also important to note that findings from these studies may differ by demographic group, particularly for those experiencing significant psychosocial stress. Future studies should focus on including diverse populations and adopting longitudinal designs to capture the dynamic nature of menopausal experiences. Finally, in future research, the Adverse Childhood Experience measure [ 51 ], administered retrospectively, could be very useful in further examining the associations between chronic psychosocial stress and long-term health outcomes related to menopausal transition. Investigating these relationships may provide deeper insights into how early life stressors contribute to the severity of menopausal symptoms and overall health in later life, further establishing the usefulness of the life course theory in this work.

Limitations

While this scoping review provides valuable insights, it is essential to acknowledge its limitations. The exclusion of non-English articles and reliance on specific databases may have omitted relevant studies. Additionally, the scoping review approach does not evaluate the quality of individual studies. To address these limitations, future research could undertake systematic reviews or meta-analyses, and include articles published in other languages, to provide more robust evidence.

The life-altering transition of menopause, and its precursor perimenopause, considerably impact health, well-being, and overall quality of life. Therefore, it is critical to explore how this transition varies for individuals who have experienced chronic psychosocial stress and those with other experiences such as intimate partner violence or racialization as Black. Concepts and frameworks of health equity, racial and social justice, and human rights should be used to understand and address the problem of health care fragmentation impacting menopause. This means looking at both the upstream and downstream effects to understand solutions. Upstream oppression and racism impact health and the ability of the stress response system to respond effectively to challenges. Gaps in care systems, along with real barriers to access, make it challenging to achieve optimal health. This scoping review, guided by the theoretical frameworks of LCT and intersectionality, included several studies investigating the transition to menopause, extracting the essential findings, highlighting geographical bias, and identifying the need for future comprehensive study designs.

The review illustrated the diverse experiences of menopausal transition among populations experiencing psychosocial stress. The findings emphasize the significance of considering social, cultural, and environmental factors in understanding menopausal experiences and tackling health disparities. Policymakers, healthcare providers, and public health practitioners should consider the identified factors and experiences to develop targeted interventions and support strategies that improve the well-being and quality of life of populations navigating the menopausal transition alongside psychosocial stressors.

Data availability

No data were generated or analyzed in this study. This manuscript is a scoping review that synthesizes existing literature; therefore, there are no primary research data to share.

McKinlay SM. The normal menopause transition: an overview. Maturitas. 1996;23(2):137–45.

Article   CAS   PubMed   Google Scholar  

Mayo Clinic [Internet]. [cited 2024 May 17]. Perimenopause - Symptoms and causes. https://www.mayoclinic.org/diseases-conditions/perimenopause/symptoms-causes/syc-20354666

Lisabeth L, Bushnell C. Menopause and stroke: an epidemiologic review. Lancet Neurol. 2012;11(1):82.

Article   PubMed   PubMed Central   Google Scholar  

Prabakaran S, Schwartz A, Lundberg G. Cardiovascular risk in menopausal women and our evolving understanding of menopausal hormone therapy: risks, benefits, and current guidelines for use. Ther Adv Endocrinol Metab. 2021;12:20420188211013917.

El Khoudary SR, Aggarwal B, Beckie TM, Hodis HN, Johnson AE, Langer RD, et al. Menopause transition and cardiovascular disease risk: implications for timing of early prevention: a scientific statement from the American Heart Association. Circulation. 2020;142(25):e506–32.

Article   PubMed   Google Scholar  

Macpherson BE, Quinton ND. Menopause and healthcare professional education: a scoping review. Maturitas. 2022;166:89–95.

Gannon L, Stevens J. Portraits of menopause in the mass media. Women Health. 1998;27(3):1–15.

Chalouhi S, Menopause. A complex and controversial journey. Post Reprod Health. 2017;23(3):128–31.

Cortés YI, Marginean V. Key factors in menopause health disparities and inequities: beyond race and ethnicity. Curr Opin Endocr Metab Res. 2022;26:100389.

Article   Google Scholar  

Faleschini S, Tiemeier H, Rifas-Shiman SL, Rich-Edwards J, Joffe H, Perng W, et al. Longitudinal associations of psychosocial stressors with menopausal symptoms and well-being among women in midlife. Menopause. 2022;29(11):1247–53.

Richard-Davis G, Wellons M. Racial and ethnic differences in the physiology and clinical symptoms of menopause. In Thieme Medical; 2013. pp. 380–6.

Gibson CJ, Huang AJ, Mccaw B, Subak LL, Thom DH, Van Den Eeden SK. Associations of intimate Partner violence, sexual assault, and posttraumatic stress disorder with menopause symptoms among midlife and older women. JAMA Intern Med. 2019;179(1):80–7.

De Mello A, Chavez A, Mukarram M, Buras MR, Kling JM. Menopausal symptoms in the Southwest United States: a cross-sectional survey of women from areas with different socioeconomic resources. Maturitas. 2021;154:7–12.

Forman MR, Mangini LD, Thelus-Jean R, Hayward MD. Life-course origins of the ages at menarche and menopause. Adolesc Health Med Ther. 2013;1–21.

Henderson KD, Bernstein L, Henderson B, Kolonel L, Pike MC. Predictors of the timing of natural menopause in the multiethnic cohort study. Am J Epidemiol. 2008;167(11):1287–94.

Bauld R, Brown RF. Stress, psychological distress, psychosocial factors, menopause symptoms and physical health in women. Maturitas. 2009;62(2):160–5.

Arnot M, Emmott EH, Mace R. The relationship between social support, stressful events, and menopause symptoms. PLoS ONE. 2021;16(1):e0245444.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Collins PH. Black feminist thought: knowledge, consciousness, and the politics of empowerment. Routledge; 2021.

Moraga C, Anzaldúa G. This bridge called my back: writings by radical women of color. State University of New York; 2022.

Crenshaw KW. On intersectionality: essential writings. New; 2017.

Halfon N, Forrest CB, Lerner RM, Faustman EM. Handbook of life course health development. 2018.

Lu MC, Verbiest S, Dominguez TP. 1. Life Course Theory: An Overview. In: Moving Life Course Theory Into Action: Making Change Happen [Internet]. American Public Health Association; 2018 [cited 2024 May 17]. https://doi.org/10.2105/9780875532967ch01

Fine A, Kotelchuck M, Rethinking MCH. The life course model as an organizing framework. US Dep Health Hum Serv Health Resour Serv Adm Matern Child Health Bur. 2010;1–20.

Elder Jr. GH. Life Course Perspective. In: The Blackwell Encyclopedia of Sociology [Internet]. 2007 [cited 2024 May 18]. https://doi.org/10.1002/9781405165518.wbeosl046

Bazzano AN, Pies C, Lu MC, Parthasarathy P, Fine A, Kotelchuck M. Development and Unfolding of the Life Course Movement in the Field of Maternal and Child Health: An Oral History. Matern Child Health J [Internet]. 2024 Jul 16 [cited 2024 Jul 30]; https://link.springer.com/ https://doi.org/10.1007/s10995-024-03938-y

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Peters MDJ, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. JBI Evid Implement [Internet]. 2015;13(3). https://journals.lww.com/ijebh/Fulltext/2015/09000/Guidance_for_conducting_systematic_scoping_reviews.5.aspx

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73.

Nosek M, Kennedy HP, Beyene Y, Taylor D, Gilliss C, Lee K. The effects of perceived stress and attitudes toward menopause and aging on symptoms of menopause. J Midwifery Womens Health. 2010;55(4):328–34.

Gerber MR, King MW, Pineles SL, Wiltsey-Stirman S, Bean-Mayberry B, Japuntich SJ, et al. Hormone therapy use in Women Veterans Accessing Veterans Health Administration Care: A National Cross-sectional Study. J Gen Intern Med. 2015;30(2):169–75.

Thurston RC, Bromberger JT, Joffe H, Avis NE, Hess R, Crandall CJ, et al. Beyond frequency: who is most bothered by vasomotor symptoms? Menopause. 2008;15(5):841–7.

Thomas HN, Hamm M, Hess R, Thurston RC. Changes in sexual function among midlife women: i’m older… and I’m wiser. Menopause. 2018;25(3):286–92.

Palmer JR, Rosenberg L, Wise LA, Horton NJ, Adams-Campbell LL. Onset of natural menopause in African American women. Am J Public Health. 2003;93(2):299–306.

Woods NF, Mitchell ES. Pathways to depressed mood for midlife women: observations from the Seattle Midlife Women’s Health Study. Res Nurs Health. 1997;20(2):119–29.

Avis NE, Stellato R, Crawford S, Bromberger J, Ganz P, Cain V, et al. Is there a menopausal syndrome? Menopausal status and symptoms across racial/ethnic groups. Soc Sci Med. 2001;52(3):345–56.

Bromberger JT, Meyer PM, Kravitz HM, Sommer B, Cordal A, Powell L, et al. Psychologic distress and natural menopause: a multiethnic community study. Am J Public Health. 2001;91(9):1435–42.

Sharps PW, Phillips J, Oguntimalide L, Saling J, Yun S. Knowledge, attitudes, perceptions and practices of African-American women toward menopausal health. J Natl Black Nurses Assoc JNBNA. 2003;14(2):9–15.

PubMed   Google Scholar  

Hudson AL, Taylor D, Lee KA, Gilliss CL. Symptom experience and self-care strategies among healthy, midlife African-American women. J Natl Black Nurses Assoc JNBNA. 2005;16(2):6–14.

Crenshaw K. Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics. https://chicagounbound.uchicago.edu/uclf/vol1989/iss1/8/

Thurston RC, Bromberger J, Chang Y, Goldbacher E, Brown C, Cyranowski JM, et al. Childhood abuse or neglect is associated with increased vasomotor symptom reporting among midlife women. Menopause. 2008;15(1):16–22.

Michopoulos V, Huibregtse ME, Chahine EB, Smith AK, Fonkoue IT, Maples-Keller J, et al. Association between perimenopausal age and greater posttraumatic stress disorder and depression symptoms in trauma-exposed women. Menopause. 2023;30(10):1038–44.

Bromberger JT, Matthews KA, Kuller LH, Wing RR, Meilahn EN, Plantinga P. Prospective study of the determinants of age at menopause. Am J Epidemiol. 1997;145(2):124–33.

Williams M, Richard-Davis G, Weickert A, Christensen L, Ward E, Schrager S. A review of African American women’s experiences in menopause. Menopause. 2022;29(11):1331–7.

Lewis Johnson T, Rowland LM, Ashraf MS, Clark CT, Dotson VM, Livinski AA et al. Key findings from Mental Health Research during the menopause transition for racially and ethnically Minoritized Women living in the United States: a scoping review. J Womens Health. 2023;jwh.2023.0276.

Evans CR. Modeling the intersectionality of processes in the social production of health inequalities. Soc Sci Med. 2019;226(January):249–53.

Jones NL, Gilman SE, Cheng TL, Drury SS, Hill CV, Geronimus AT. Life Course approaches to the causes of Health disparities. Am J Public Health. 2019;109(S1):S48–55.

Spencer K, Mortimer J. In. Development and socialization through the Adult Life Course. 2006. pp. 183–202.

Glazer G, Zeller R, Delumba L, Kalinyak C, Hobfoll S, Winchell J, et al. The Ohio midlife women’s study. Health Care Women Int. 2002;23(6–7):612–30.

Jones HJ, Sternberg RM, Janson SL, Lee KA. A qualitative understanding of midlife sources of stress and support in African-American women. J Natl Black Nurses Assoc JNBNA. 2016;27(1):24–30.

Avis NE, Crawford SL, Greendale G, Bromberger JT, Everson-Rose SA, Gold EB, et al. Duration of menopausal vasomotor symptoms over the menopause transition. JAMA Intern Med. 2015;175(4):531–9.

CDC. Adverse Childhood Experiences (ACEs). 2024 [cited 2024 Jul 11]. About Adverse Childhood Experiences. https://www.cdc.gov/aces/about/index.html

Download references

Acknowledgements

We extend our deepest gratitude to Laura Wright, the Research Support Librarian at Tulane University’s Rudolph Matas Library of the Health Sciences whose invaluable assistance and dedication were instrumental in conducting this scoping review.

The Tulane Center of Excellence in Maternal and Child Health, which is funded by the Health Resources and Services Administration (HRSA) of the U.S. Department of Health and Human Services (HHS) under grant number T76MC04927 and title Maternal and Child Health Public Health Training Program. This information or content and conclusions are those of the author and should not be construed as the official position or policy of, nor should any endorsements be inferred by HRSA, HHS or the U.S. Government.

Author information

Emma A. Blackson and Christina McCarthy contributed equally to this work.

Authors and Affiliations

Department of Social, Behavioral, and Population Sciences, Tulane School of Public Health and Tropical Medicine, New Orleans, LA, USA

Emma A. Blackson, Caryn Bell, Shokufeh Ramirez & Alessandra N. Bazzano

Center of Excellence in Maternal and Child Health, Tulane School of Public Health and Tropical Medicine, New Orleans, LA, USA

Emma A. Blackson, Shokufeh Ramirez & Alessandra N. Bazzano

School of Medicine, Tulane University, LA, New Orleans, United States

Christina McCarthy

You can also search for this author in PubMed   Google Scholar

Contributions

A.N.B. contributed to the conception or design of the work, provided interpretation of data analysis, and critically revised the article. E.A.B. and C.M. were responsible for data collection, data analysis and interpretation, drafting the article, and critically revising the article. C.B. and S.R. also provided an interpretation of the data analysis and critically revised the article. All authors were actively involved in interpreting the results, writing, revising, and approving the final manuscript. They have approved the submitted and ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated, resolved, and documented in the literature.

Corresponding author

Correspondence to Emma A. Blackson .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Blackson, E.A., McCarthy, C., Bell, C. et al. Experiences of menopausal transition among populations exposed to chronic psychosocial stress in the United States: a scoping review. BMC Women's Health 24 , 487 (2024). https://doi.org/10.1186/s12905-024-03329-z

Download citation

Received : 10 June 2024

Accepted : 23 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1186/s12905-024-03329-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Discrimination
• Psychological
• Women’s health
• Postmenopause
• Premenopause
• Systemic racism
• Stress disorders

BMC Women's Health

ISSN: 1472-6874

• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 03 September 2024

Evaluating coupling coordination between urban smart performance and low-carbon level in China’s pilot cities with mixed methods

• Xiongwei Zhu 1 ,
• Dezhi Li 1 , 2 ,
• Shenghua Zhou 1 ,
• Shiyao Zhu 3 &
• Lugang Yu 1  

Scientific Reports volume  14 , Article number:  20461 ( 2024 ) Cite this article

Metrics details

• Climate-change adaptation
• Climate-change impacts
• Environmental impact
• Sustainability

The construction models of smart cities and low-carbon cities are crucial for advancing global urbanization, enhancing urban governance, and addressing major urban challenges. Despite significant advancements in smart and low-carbon city research, a consensus on their coupling coordination remains elusive. This study employs mixed-method research, combining qualitative and quantitative analyses, to investigate the coupling coordination between urban smart performance (SCP) and low-carbon level (LCL) across 52 typical smart and low-carbon pilot cities in China. Independent evaluation models for SCP and LCL qualitatively assess the current state of smart and low-carbon city construction. Additionally, an Entropy–TOPSIS–Pearson correlation–Coupling coordination degree (ETPC) analysis model quantitatively examines their relationship. The results reveal that smart city initiatives in China significantly outperform low-carbon city development, with notable disparities in SCP and LCL between eastern, non-resource-based, and central cities versus western, resource-dependent, and peripheral cities. A strong positive correlation exists between urban SCP and overall LCL, with significant correlations in management, society, and economy, and moderate to weak correlations in environmental quality and culture. As SCP levels improve, the coupling coordination degree between the urban SCP and LCL systems also increases, driven primarily by economic, management, and societal factors. Conversely, the subsystems of low-carbon culture and environmental quality show poorer integration. Based on these findings, this study proposes an evaluation system for smart and low-carbon coupling coordination development, outlining pathways for future development from the perspective of urban complex systems.

Similar content being viewed by others

An empirical analysis of the coupling and coordinated development of new urbanization and ecological welfare performance in China’s Chengdu–Chongqing economic circle

Decomposing the comprehensive efficiency of major cities into divisions on governance, ICT and sustainability: network slack-based measure model

Impact of urban spatial structure elements on carbon emissions efficiency in growing megacities: the case of Chengdu

Introduction.

Cities, as centers of population and economy, play crucial roles in cultural exchange, social integration, transportation, communication, and disaster response in modern societal development 1 , 2 . According to the United Nations Human Settlements program’s “2022 World Cities Report”, as of 2021, the global urbanization rate has reached 56%, and it is projected that by 2050, an additional 2.2 billion people will live in cities, increasing the urbanization rate to 68% 3 . North America and European countries are approaching urbanization saturation, with little fluctuation expected, while urbanization in Asia and Africa will accelerate notably 4 . Particularly in China, the world’s second-largest economy, as of 2022, the urbanization rate is only 64.7%, ranking 96th globally, indicating significant potential for growth compared to developed countries like the USA and the UK 5 . The Chinese government places high importance on urbanization development. It was clearly stated in the “2020 State Council Government Work Report” that new urbanization is a key measure for achieving China’s modernization. Moreover, in the “14th Five-Year Plan (2021–2025) and the Long-Range Objectives Through the Year 2035”, detailed strategies are outlined for optimizing the urban layout and promoting urban–rural integration, among other policies to advance urbanization 6 . However, urbanization, as a process of continuous concentration of population and industrial elements in cities, while bringing opportunities for economic growth and social development, also presents a series of challenges such as environmental pressure, resource constraints, and increased demand for services 7 , 8 .

In 2008, the American company IBM introduced the concept of a “Smart Planet”, which garnered widespread attention globally 9 . The concept of a smart city, as a specific application within this framework, aims to enhance urban management and service efficiency through the integration and innovative application of Information and Communication Technology (ICT), thereby improving the quality of life for residents, optimizing resource use, reducing environmental impact, and promoting economic development and social progress 10 , 11 . Currently, the smart city construction model is seen as one of the effective means to advance global urbanization, improve urban governance, and solve major urban issues 12 . In 2009, IBM released the “Smart Planet: Winning in China” plan, outlining China’s five major thematic tasks in constructing a “Smart Planet” (sustainable economic development, corporate competitiveness, energy efficiency, environmental protection, and social harmony) 13 . The construction of smart cities, as a key measure to achieve these thematic tasks, has received significant attention from the Chinese government. In 2014, the Chinese government elevated smart city construction to a “national strategy”, considering it a cornerstone of China’s future economic and urban development strategies. By 2016, over 500 Chinese cities had initiated or announced smart city pilot construction plans, accounting for nearly half of all such projects planned or underway globally 14 . In recent years, with the continuous release of policy benefits related to smart city construction in China and substantial capital investment, China has become a leader in driving global smart city initiatives 15 . However, an undeniable fact is that while smart city construction models promote economic development and improve the quality of life for residents, the new infrastructure supporting the operation of smart cities, such as big data centers, 5G shared base stations, and Beidou ground-based augmentation stations, result in substantial energy consumption and significant carbon emissions 16 . Research shows that in 2018, the total electricity consumption of data centers in China supporting IT infrastructure reached 160.9 billion kilowatt-hours, exceeding the total electricity consumption of Shanghai for that year and accounting for about 2% of China’s total electricity consumption, with carbon emissions nearing 100 million tons 17 . The Environmental Defense Fund (EDF) predicts that by 2035, the total electricity consumption of China’s data centers and 5G base stations will reach 695.1–782 billion kilowatt-hours, accounting for 5–7% of China’s total electricity consumption, with total carbon emissions reaching 230–310 million tons 18 .

In 2022, global energy-related CO 2 emissions increased by 0.9%, reaching a record high of over 36.8 Gt. Concurrently, atmospheric CO 2 concentrations continued to rise, averaging 417.06 parts per million, marking the eleventh consecutive year with an increase exceeding 2 ppm 19 . According to the World Meteorological Organization (WMO), the global surface temperature in September 2023 was 1.44 °C higher than the twentieth century average, setting a new historical record 20 . The continuous rise in global temperatures has led to frequent occurrences of disastrous events such as extreme heat, torrential rains, floods, forest fires, and hurricanes in recent years, causing significant loss of life and property damage 21 . World Health Organization (WHO) data indicates that in 2022, there were at least 29 weather disaster events globally causing billions of dollars in losses, with approximately 61,672 deaths in Europe due to heatwave-related causes 22 . As global climate issues become increasingly severe, the call for global carbon emission reduction is growing louder. Cities, as highly concentrated areas of population and economic activities, according to the Global Report by the United Nations Human Settlements Programme (UN-Habitat), consume 60–80% of the global energy and contribute to over 75% of global CO 2 emissions 23 . As the largest global emitter of carbon, China’s CO 2 emissions in 2022 accounted for 27% of the global total 24 . Given China’s influence in the global economy, technological innovation, and international cooperation, international organizations and global climate policies generally believe that China’s efforts in carbon reduction are crucial to achieving the global 1.5 °C climate goal 25 . In recent years, the Chinese government has actively promoted the construction of low-carbon pilot cities. To date, three batches of low-carbon pilot cities have been implemented in China, bringing the total number of such cities to 81 26 .

However, the report “China’s Digital Infrastructure Decarburization Path: Data Centers and 5G Carbon Reduction Potential and Challenges (2020–2035)” indicates that compared to peak carbon emissions expected around 2025 in key sectors like steel, building materials, and non-ferrous metals in China, the “lock-in effect” of carbon emissions from digital infrastructure poses a significant challenge to achieving China’s peak carbon and carbon neutrality goals 27 , 28 , 29 . Given the urgency of global climate change, it raises the question of the correlation between smart cities and low-carbon cities: is it positive, negative, or non-existent? Should the pace of smart city development be slowed to achieve sustainable urban development goals, considering the significant carbon dioxide emissions resulting from current technological choices, social habits, and policy frameworks? To address these practical issues, it is first essential to conduct an objective and accurate assessment of urban SCP and LCL. However, due to the complexity and diversity of urban carbon emissions sources, current measurement and estimation techniques fail to capture all emission types. This limitation hampers the ability to obtain comprehensive, accurate, and timely city-level carbon emission data 30 , 31 . To address this challenge, this paper decomposes smart cities and low-carbon cities into their interdependent and interactive subsystems (i.e., economic, political, cultural, social, and ecological) viewed through the lens of urban complex systems. It then develops evaluation models for both city types and conducts empirical analyses in 52 representative Chinese pilot cities. Based on these analyses, the paper elucidates the coupling coordination degree between SCP and LCL and proposes a specific pathway for their coordinated development.

This paper is therefore structured as follows: “ Literature review ” section offers an overview of the relevant literature, laying the foundation for the introduction of SCP and LCL. Subsequently, SCP and LCL are identified clearly, and measurement based on a mixed method for the coupling coordination degree is established in “ Methodology ” section, followed by a case demonstration for the introduced method in “ Results ” section and the demonstration results analysis in “ Discussions and implications ” section. Finally, “ Conclusions ” section summarizes the study’s main findings and contributions, discusses its limitations, and suggests directions for future research.

Literature review

Evaluation of smart city: contents, methods, and subjects.

The evaluation of smart cities is a central research area within the smart city development field. Developing standardized evaluation criteria serves the dual purpose of defining smart city development boundaries and scientifically measuring its effectiveness. This, in turn, facilitates the achievement of development goals centered on evaluation-driven construction, improvement, and management 32 . We conducted data collection on “smart city*” AND “evaluation”, resulting in the selection of 82 articles. This involved an extensive search of the Wos Core Collection database for articles published in the period from January 2019 to January 2024.

To facilitate a clearer understanding for readers of current research on smart city evaluation, we have categorized it by evaluation contents , evaluation methods , and evaluation subjects .

Cluster1-evaluation contents (what to evaluate), including smart city evaluation dimensions and indicators. By analyzing the article content, it’s clear that most smart city evaluation approaches align with six core dimensions: economy, quality of life, governance, people, mobility, and environment 13 , 15 . Centered around these six dimensions, international organizations (ISO, ETSI, UN, and ITU) and scholars have established various sets of smart city evaluation indicators, considering the interdependencies among urban economic, environmental, and social factors, all in alignment with the goals of sustainable urban development 32 , 33 , 34 . Notably, Sharifi 35 compiled a comprehensive list of indicators incorporating a wide range of assessment schemes. This list not only covers the scope of the evaluation indicators (project/community/city) and their data types (primary/secondary) but also considers the stages of smart city development (planning/operation) and stakeholder involvement 36 . Subsequent research predominantly utilizes the same criteria as Sharifi 35 to identify indicator sets, taking into account the specific needs of each city and defining the spatial and temporal scales of the indicator sets 37 .

Cluster 2-evaluation methods (How to evaluate) , including smart city evaluation methods and tools. Research in this field focuses on three main areas: identifying evaluation indicators for smart cities, computing composite index, and developing evaluation models 38 , 39 . Methods for indicator identification mainly include literature review, case studies, brainstorming, the Delphi method, and data-driven techniques 40 , 41 . The Analytic Hierarchy Process (AHP) is commonly used for calculating composite indices, yet it faces issues like subjective biases and data size limitations 42 . Alternative methods, such as the Analytical Network Process (ANP) and the Decision-Making Trial and Evaluation Laboratory (DEMATEL), are used to address these drawbacks by simulating inter-indicator interactions. Additionally, techniques like Principal Component Analysis (PCA) and Data Envelopment Analysis (DEA) are applied for indicator weighting. Finally, smart city evaluation models are constructed to aggregate various dimensions and indicators into a unified score, facilitating project comparison and ranking, and highlighting areas needing improvement 43 , 44 .

Cluster 3-evaluation subjects (Who performs the evaluation) , including smart city stakeholders and participants. Smart city evaluations involve various stakeholders and participants. These complex processes see each entity, including government agencies, international organizations, academic institutions, industry sectors, and NGOs, contributing to the smart cities’ planning, development, and management 45 , 46 . Key organizations in this realm are the International Organization for Standardization (ISO), International Telecommunication Union (ITU), United Nations Human Settlements Programme (UN-Habitat), Smart Cities Council, European Institute of Innovation and Technology (EIT Urban Mobility), and World Council on City Data (WCCD). Additionally, numerous countries have established their own smart city evaluation standards to direct and review smart city progress 11 . Notable examples are the “One New York: The Plan for a Strong and Just City” in the USA, the “BSI PAS 180” in the UK, Singapore's “Smart Nation Initiative”, and China’s “National New-type Smart City Evaluation Indicator System”.

Evaluation of low-carbon city: contents, methods, and subjects

As more countries integrate low-carbon city development into their national strategies and plans, conducting scientific evaluations of cities’ current low-carbon development levels to encourage them to adopt corresponding measures for improvement has become a key strategy in advancing cities towards a low-carbon future 47 . In the Wos Core Collection database, we conducted a search for studies spanning January 2018 to January 2023 with “low-carbon city*” AND “evaluation” as keywords, subsequently identifying 98 pertinent articles through two rounds of screening.

This section, maintaining the research framework of “ Evaluation of smart city: contents, methods, and subjects ” section ( evaluation contents, methods, and subjects ), organizes low-carbon city research to enable comparison with smart city evaluations.

Cluster 1-evaluation contents (what to evaluate), including low-carbon city evaluation systems, dimensions, and indicators. Current research focusing on low-carbon cities primarily spans six key domains: urban low-carbon scale, energy, behavior, policy, mobility, and carbon sinks. The evaluation dimensions for low-carbon cities are mainly divided into two types: single-criterion systems concentrating on specific low-carbon aspects (such as low-carbon economy, low-carbon energy, etc.), and comprehensive multi-criteria systems assessing the overall urban low-carbon development 48 , 49 . Compared to single-criterion evaluation systems, comprehensive and multi-criteria evaluation systems are increasingly gaining attention from scholars. These scholars share the view that low-carbon city construction is a diverse, dynamic, interconnected process that requires comprehensive consideration of various urban aspects, including economy, society, and environment, and involves coordinating the actions of different stakeholders to achieve sustainable urban development 50 , 51 . Additionally, international institutions and many national governments have also published low-carbon city evaluation frameworks from the perspective of comprehensive and multi-criteria evaluation systems. The most notable examples include the United Nations Commission on Sustainable Development, which set 30 indicators from four dimensions: social, environmental, economic, and institutional, to evaluate the level of urban low-carbon development. The Chinese Academy of Social Sciences proposed the “China Low Carbon City Indicator System”, covering 8 dimensions such as economy, energy, facilities, and 25 specific indicators including energy intensity, per capita carbon emissions, and forest coverage rate.

Cluster 2-evaluation methods (How to evaluate) , including low-carbon city evaluation methods and tools. Firstly, identifying evaluation indicators as the initial step in constructing a low-carbon city evaluation model, current research methods not only include traditional methods like literature review and expert interviews but also increasingly involve scholars using dynamic perspectives based on urban complex systems, applying models like DPSR (Driving forces-Pressures-State-Response), STIRPA (Stochastic Impacts by Regression on Population, Affluence, and Technology), the Environmental Kuznets Curve (EKC), and STEEP (Social, Technological, Economic, Ecological, and Political) for indicator identification 52 , 53 . Secondly, weighting evaluation indicators, an essential part of model construction, typically involves methods like subjective weighting (expert scoring, Delphi method, AHP) 54 , objective weighting (PCA, Entropy weight method, variance analysis), and combined weighting (DEA) 55 . Each method has its characteristics and suitable scenarios and should be selected according to specific circumstances. Additionally, quantitative assessment of regional carbon emissions using methods like carbon footprint analysis, baseline emission comparison, and Life Cycle Assessment (LCA) is also becoming a research focus 56 .

Cluster 3-evaluation subjects (Who performs the evaluation) , including low-carbon city stakeholders and participants. The evaluation of low-carbon cities also involves multiple stakeholders (government, enterprises, residents, etc.) 57 . Among them, international organizations like the International Organization for Standardization (ISO), the International Energy Agency (IEA), and the World Meteorological Organization (WMO) have played significant roles in establishing low-carbon city evaluation standards and promoting global low-carbon city development. Additionally, due to economic, policy, and perception factors, current low-carbon city construction relies primarily on government financial input, with social capital and public participation in low-carbon city construction noticeably lacking 58 . Therefore, how to enhance the awareness of enterprises and residents as main actors in low-carbon city construction has become a current research focus.

Coupling coordination analysis between SCP and LCL

Smart cities and low-carbon cities, as important urban development models for the future, have seen an increasing focus on their interrelation by scholars in recent years, becoming an emerging research hotspot in the field. In the Wos Core Collection database, we searched for studies from January 2018 to January 2024 using the keywords “smart city*” “low-carbon city*” “correlation analysis” “coupling coordination analysis” and “urban sustainability”. After two rounds of screening, 24 related studies were selected for analysis.

From the perspective of research results, the current research conclusions about the correlation between low-carbon cities and smart cities primarily include two main points: (i) SCP and LCL cannot achieve coupling coordination development. Some scholars argue that SCP and LCL differ in their focus: SCP emphasizes urban technological and economic development, while LCL focuses more on urban ecological construction 17 . Particularly, De Jong identified 12 urban development concepts, including smart city, low-carbon city, eco-city, and green city. He believes that a clear distinction must be made in the conceptual definition of these types of cities to more accurately guide future urban planning 59 . Furthermore, some scholars argue that the relationship between SMC and LCC is negatively correlated. Deakin believes that the direct environmental benefits of IoT technology are insufficient to achieve urban sustainability goals 60 . Barr et al. argue that the logic of smart cities often leads city administrations to prioritize superficial changes and promote individual behavioral shifts, detracting from the crucial task of reconfiguring urban infrastructure for low-carbon lifestyles 61 , 62 . (ii) SCP and LCL can achieve coupling coordination development. Some scholars believe there is a positive correlation between SCP and LCL, with SCP potentially promoting the development of LCL. Specifically, the intelligent systems built by SCP can effectively match urban energy supply and demand, reducing urban carbon emissions, such as through smart grids and intelligent transportation networks 18 . It is worth noting that most of the studies on the coupling coordination relationship between urban SCP and LCL are based on perspectives of individual urban subsystems such as technology, economy, management, industrial structure, and society. They lack a comprehensive consideration of the city as a complex system 59 , 61 , 63 .

From the perspective of research methodologies, coupling coordination analysis is a fundamental statistical approach for examining relationships between two or more variables. This analysis typically employs techniques such as Pearson’s correlation coefficient, Spearman’s rank correlation coefficient, Kendall’s tau, partial correlation, point-biserial correlation, and multiple correlations. Each technique offers unique insights into the nature and strength of the interdependencies among variables 61 . The selection of an appropriate method depends on the data type (continuous, ordinal, or categorical), its distribution (e.g., normal distribution), and the specific objectives of the research.

In summary, although existing research has made significant contributions to the independent evaluation and advancement of smart cities and low-carbon cities, including their relevant construction content, main actors, as well as some specific measures such as empowering cities with data intelligence for low-carbon economic development and transitioning industrial structure to low-carbon, there are still some important knowledge gaps. On the one hand, current research primarily analyzes the coupling coordination relationship between urban SCP and LCL from the micro-perspective of individual urban subsystems such as economic and energy systems. This approach lacks a macroscopic perspective from the complex urban system, which is detrimental to the comprehensive development of cities 60 , 64 , 65 . On the other hand, current studies often only conduct basic qualitative comparisons of the relationship between the development levels of urban SCP and LCL from a quantitative or qualitative perspective. They lack a comprehensive analytical approach that integrates both qualitative and quantitative analyses for further exploration of the coupling coordination relationship between urban SCP and LCL. This shortfall hinders the sustainable development of cities.

To fill these knowledge gaps, this study employs a mixed-methods approach, combining qualitative and quantitative analyses, to examine the model of coupling coordination between urban SCP and LCL. It also develops recommendations to enhance this coupling coordination, aiming to support sustainable development goals. Furthermore, this research selects 52 typical low-carbon and smart pilot cities in China as case studies, ensuring both scientific validity and practical applicability of the findings. Additionally, to enhance the logical coherence and readability of this study, we posit that a coupling coordination relationship exists between urban SCP and LCL and thus propose Hypothesis 1 .

Hypothesis 1

There is a substantial degree of coupling coordination between the overall urban system’s SCP and LCL, yet there are disparities in this coordination degree among the subsystems of economy, society, politics, culture, and ecology.

Methodology

Research framework.

The construction of low-carbon and smart cities, as key pathways to urban sustainability, necessitates examining their interplay and fostering their collaborative development for achieving sustainability goals 66 . This research employs a sequential framework, including Conceptual, Data, Analysis, and Decision-making Layers, to methodically explore the coupling coordination relationship between SCP and LCL, with the framework illustrated in Fig.  1 .

Research framework.

Firstly , in the Conceptual Layer, this study aligns with the United Nations’ objectives for sustainable cities, encompassing economic growth, social equity, better life conditions, and improved urban environments. Integrating these with China’s “Five-Sphere Integrated Plan (economy, politics, culture, society, and ecological environment construction)” for urban development, the research dissects the components of smart city systems (such as information infrastructure, information security, public welfare services) and low-carbon city systems (including low-carbon construction, transportation, and industry), with the aim to collect indicators. Secondly , in the Data Layer, this research develops smart city and low-carbon city evaluation systems, grounded in national standards and official statistics, to qualitatively examine the correlation between SCP and LCL from a macro perspective. Thirdly, in the Analysis Layer, this study selects 52 cities, both smart and low-carbon pilot cities in China, as samples for quantitative analysis. The process involves standardizing indicators, scoring and ranking the cities based on their smart performance and low-carbon levels, followed by employing Pearson’s correlation coefficient and coupling coordination degree model to scientifically analyze the correlation between SCP and LCL. Finally, in the Decision-making Layer, the study examines the coupling coordination relationship between urban smart performance, the overall low-carbon level, and the low-carbon level across five dimensions, which is key for us to test Hypothesis 1 . It also formulates development paths for the coupling coordination of smart and low-carbon cities.

SCP index system construction

Since the concept of smart cities was introduced in 2008, many national governments have established smart city evaluation standards. Due to varying national conditions, SCP evaluation indicators differ across countries. As the sample cities in this study are Chinese smart pilot cities, the selection of SCP evaluation indicators primarily references relevant Chinese national standards. As a global pioneer in smart city development, China released the “Evaluation indicators for new-type smart cities (GB/T 33356-2016)” in 2016 and revised it in 2022. This national standard, with its evaluative indicators, clearly defines the key construction content and development direction of new smart cities, aiming to specifically enhance the effectiveness and level of smart city construction, gaining significant recognition within the industry.

This study, grounded in the concept of a city’s “Five-in-One” sustainable development, is guided by three principles of “Inclusive well-being & Ecological harmony”, “Digital space & Physical space”, and “New IT technologies & Comprehensive services”. It also adheres to the “people-oriented concept” and adopts an “urban complex dynamic perspective” in the process of smart city construction. Additionally, it follows the principle of “similar attributes of evaluation objects”. Based on these foundations, the study establishes three criteria for selecting evaluation indicators, including scientific, coordination, and representation. Drawing on the Chinese government’s smart city evaluation standards and utilizing a literature review methodology, this research constructs an SCP evaluation indicator system for cities, as detailed in Supplementary Appendix Table A1 . The SCP index system includes six primary indicators, including smart public service (SPE), precise governance (PG), information infrastructure (II), digital economy (DE), innovative development environment (IDE), and citizen satisfaction (SCS). It also features 24 secondary indicators, such as traffic information services, grassroots smart governance, and spatio-temporal information platforms. Importantly, to explore the correlation between smart cities and low-carbon cities more effectively, the study deliberately omits “Internet + Green Ecology” related indicators from the smart city evaluation system. To ensure the accuracy and representativeness of these indicators, they were validated through expert consultation, public participation, and comprehensive statistical methods.

LCL index system construction

Current international organizations and academic perspectives on low-carbon city evaluation systems are predominantly based on the urban complex systems approach, considering the interplay and interaction of aspects such as low-carbon society, economy, and technology. Consistent with the principles for selecting SCP evaluation indicators, the choice of LCL evaluation indicators in this study primarily adheres to relevant Chinese national standards and related literature.

As a proactive practitioner in global low-carbon city development, in 2021, the Chinese government released the “Sustainable Cities and Communities—Guides for low-carbon development evaluation (GB/T 41152-2021)”. This national standard evaluates the level of urban low-carbon development, clarifying the key directions for such development, and serves as a current guide for low-carbon city construction in China. Thus, this study, grounded in the “Five-in-One” sustainable urban development framework and guided by the principles of “carbon reduction & pollution reduction”, “green economic growth”, and “enhanced carbon sequestration capacity”, combines the previously established principles of scientific, coordination, and representative for selecting evaluation indicators. It establishes an LCL index system based on the Chinese government’s evaluation standards and relevant literature. Specifically, the LCL evaluation index system constructed in this study includes five primary indicators, including low-carbon economic (LCE), low-carbon society (LCS), low-carbon environmental quality (LCEQ), low-carbon management (LCM), and low-carbon culture (LCC), as well as 22 secondary indicators such as energy consumption per unit of GDP and carbon emission intensity, as shown in Supplementary Appendix Table A2 . Similarly, to ensure the accuracy and representativeness of the indicators, the specific indicators were validated through expert consultation, public participation, and comprehensive statistical methods.

Analysis model construction

In this study, an Entropy-TOPSIS-Pearson correlation-Coupling coordination degree (ETPC) analysis model is constructed to quantitatively analyze the coupling coordination relationship between Urban SCP and LCL. The entropy method is first applied for objective weighting of evaluation indices, ensuring data objectivity and reducing subjective bias, thus enhancing the model’s accuracy and fairness. Next, the TOPSIS method is used to rank sample cities based on their smart performance and low-carbon levels, providing a straightforward and intuitive ranking mechanism. The Pearson correlation method then examines the correlation between SCP and LCL, offering data-driven insights into the dynamic relationships between these variables. Finally, the coupling coordination model calculates the degree of coordination between SCP and LCL, providing a theoretical basis for subsequent enhancement pathways and policy recommendations. The ETPC model constructed in this study has several advantages and complementarities, allowing for a comprehensive analysis and evaluation of the research question from various perspectives. Additionally, the ETPC model can be broadly applied to other multidimensional evaluation and decision analysis issues, such as the coupling coordination between various public health interventions and community health levels, and the comprehensive effects of different economic policies on regional economic development and environmental impact. Specific analysis steps are outlined as follows.

Step 1: Conduct the data normalization process.

where x ij and y ij represent respectively the original and standardized value for the indicator j in referring to the sample case i ( i  = 1,2,3,…, m; j  = 1,2,3,…, n ), max (x j ) and min (x j ) denote respectively the largest and smallest value among all m samples for the indicator j , P ij represents the value proportion of indicator j in the sample case i to the summation value of the indicator from all cases.

Step 2: Calculate the weight and measure the comprehensive level based on entropy method.

The entropy weight method, an objective approach deriving weights from sample characteristics, mitigates expert bias, enhancing the objectivity and credibility of indicator weighting 67 . This study employs this method, determining weights through the calculation of each indicator’s information entropy, and measure the comprehensive level of the subsystem.

where m is the total number of sample cases, \({e}_{j}\) demonstrates the entropy value of the j indicator and \({\omega }_{j}\) denotes the weight of indicator j , and V represent the comprehensive level.

Step 3: Conduct a ranking of evaluation objects based on TOPSIS method.

A key limitation of the entropy method is its tendency to neglect the significance of indicators. The TOPSIS method, addressing this issue, is an ideal-solution-based ranking technique that aids in multi-objective decision-making among finite options 68 . In this approach, the study first determines positive and negative ideal solutions, measures each objective’s distance to these ideals, and subsequently ranks the subjects by the proximity of each objective to the ideal solution.

where \({ V}^{+}\) and \({V}^{-}\) respectively represent the best ideal solution and the worst ideal solution, \({D}_{i}^{+}\) and \({D}_{i}^{-}\) represent the distances from the objective to the positive and negative ideal solutions, respectively. \({C}_{i}\) indicates the closeness of the evaluation objective to the optimal solution, with \({C}_{i}\in \left[\text{0,1}\right]\) . A larger \({C}_{i}\) value suggests stronger smart and low-carbon development capabilities of the sample city.

Step 4: Analyze the correlation based on Pearson correlation method.

The Pearson correlation method is commonly used to measure the correlation coefficient between two continuous random variables, thereby assessing the degree of correlation between them 69 . In this study, based on the results from Steps 1–3, two sets of data are obtained representing the smart development level and low-carbon development level of sample cities, \(A:\left\{{A}_{1},{A}_{2},\dots ,{A}_{n}\right\}\) and \(B:\left\{{B}_{1},{B}_{2},\dots ,{B}_{n}\right\}\) . The overall means and covariance of both data sets are calculated, resulting in the Pearson correlation coefficient between the two variables.

where \({A}_{i}\) and \({B}_{i}\) respectively represent the SCP and LCL of sample cities. \(E\left(A\right)\) and \(E\left(B\right)\) are the overall means of the two data sets, \({\sigma }_{A}\text{ and }{\sigma }_{B}\) are their respective standard deviations, \(cov(A,B)\) is the covariance, and \({\rho }_{AB}\) is the Pearson correlation coefficient. When the correlation coefficient approaches 0, the relationship weakens, as it nears − 1 or + 1, the correlation strengthens.

Step 5: Analyze the coupling coordination degree based on the coupling coordination model.

The coupling coordination degree characterizes the level of interaction between different systems and serves as a scientific model for measuring the coordinated development level of multiple subsystems or elements 70 . This study has developed a model to measure the coupling coordination degree between two systems.

where C defines the coupling degree, \({f}_{1}\) and \({f}_{2}\) are the evaluation values of SCP and LCL respectively. CPD represents the coupling coordination degree. \(\alpha\) , \(\beta\) are the coefficient to be determined, indicating the importance of the systems. This study assumes that each system is equally important. Thus \(\alpha =\beta =1/2.\)

In this study, building upon the framework established by a preceding study, a classification system for the coupling coordination degree was developed. This system delineates the various types of coupling-coordinated development among SCP, LCL, LCS, LCM, LCEQ, and LCC. Current research on the division of coupling coordination degree intervals often uses an average distribution within the [0, 1] range 70 . However, due to the large sample size and the wide distribution range of coupling coordination degrees in this study, we have categorized these types into ten distinct levels based on their rank, as detailed in Table 1 .

Selection of sample cities and data collection

The Chinese government has prioritized the development of smart and low-carbon cities. Since 2010, it has launched 290 smart city pilots and 81 low-carbon city pilots across various regions, reflecting different levels of development, resource allocations, and operational foundations. To maintain the scientific integrity of our study, we established stringent criteria for selecting sample cities: (i) each city must be concurrently identified as both a smart and a low-carbon city pilot, and (ii) their government agencies must have issued data on key performance indicators for these initiatives. Following these criteria, our research has ultimately selected 52 cities as samples, as detailed in Fig.  2 . It is noteworthy that these 52 typical case cities are almost all provincial capitals in China, mostly located within the Yangtze River Delta, Pearl River Delta, Jingjinji (Beijing–Tianjin–Hebei), and Western Triangle economic regions. Additionally, according to the “Globalization and World Cities Research Network (GaWC) World Cities Roster 2022 (GaWC2022)”, these cities are ranked within the top 200 globally. Therefore, given the scope of this research, these case cities offer significant representativeness and can serve as valuable models for promoting development in other urban areas. The data for this paper were sourced from the “China Low-Carbon Yearbook (2010–2023)”, the “China Environmental Statistics Yearbook (2010–2023)”, and low-carbon city data published by the governments of the sample cities. Additionally, this study addressed any missing data by averaging the data from adjacent years and applying exponential smoothing.

52 sample cities and their geographic locations.

Weighting values between evaluation indicators

The entropy weighting values between the 20 indicators of SCP and the 19 indicators of LCL are calculated by applying the data described in “ Weighting values between evaluation indicators ” section to formula ( 1 )–( 5 ), and the results are shown in Supplementary Appendix Tables A3 and A4 . Specifically, within the SCP evaluation framework, SPE and II are assigned the highest weights, while LCS and LCM are allocated the highest weights within the LCL evaluation framework. Conversely, SCS and LCC have attributed the lowest weights in their respective contexts.

Evaluation of SCP and LCL in sample cities

Utilizing the data from “ Selection of sample cities and data collection ” section and the weighting values derived in “ Weighting values between evaluation indicators ” section, we can determine the SCP and LCL of sample cities using the TOPSIS method, as outlined in formulas ( 6 )–( 9 ). The results are illustrated in Supplementary Appendix Table A5 and Fig.  3 . In this study, the value of the closeness coefficient (C i ) is used to indicate the relative closeness of a particular sample city to the negative ideal point 71 . The negative ideal point represents the worst solution of the ideal, where the individual attribute values reach their worst in each alternative. Therefore, a larger value of closeness indicates better smart city performance or a lower carbon level of a sample city 72 . C LCL and C SCP respectively represent the low-carbon level closeness coefficient and the smart city performance closeness coefficient. In referring to Supplementary Appendix Table A5 , the best three cities of SCP are Shenzhen, Shanghai, and Hangzhou, whilst the worst three cities are Yan’an, Jincheng, and Xining. Furthermore, Chengdu, Qingdao, and Beijing are the best there low-carbon level performers. Whilst Jincheng, Urumqi, and Huhehaote are the three worst.

TOPSIS-based analysis of SCP with LCL in 52 sample cities.

In referencing Fig.  3 , this study considers SCP data of sample cities as the control variable and ranks them in ascending order based on TOPSIS results. We then examine changes in LCL data to ascertain the correlation between these variables, yielding two key research conclusions: on one hand, analysis of 52 sample cities demonstrates a general ascending trend in both SCP and LCL data curves. This trend suggests a positive correlation between these two parameters. On the other hand, the LCL data, in contrast to the consistent rise in SCP, exhibits notable fluctuations and wider dispersion. This indicates that the positive correlation between SCP and LCL, while present, is not markedly robust.

Correlation results of SCP and LCL in sample cities

Correlation analysis of urban SCP and overall-LCL. This analysis employs the closeness coefficient (C i ) to assess SCP and overall-LCL in sample cities for Hypothesis 1 in Eqs. ( 10 ) and ( 11 ). The results are presented in Table 2 . Additionally, a linear regression analysis is conducted to determine the presence and magnitude of the relationship between SCP and LCL in these cities, as shown in Fig.  4 .

The scatter and regression of SCP and LCL: ( A ) SCP & Overall-LCL; ( B ) SCP & LCM; ( C ) SCP & LCS; ( D ) SCP & LCE; ( E ) SCP & LCQE; ( F ) SCP & LCC.

Considering the closeness coefficient range, correlation is categorized into five levels: very weak ( \(\left|{\rho }_{AB}\right|<0\) .1), weak ( \(0.1\le \left|{\rho }_{AB}\right|<0\) .3), moderate ( \(0.3\le \left|{\rho }_{AB}\right|<0\) .5), strong ( \(0.5\le \left|{\rho }_{AB}\right|<0\) .7), and very strong ( \(0.7\le \left|{\rho }_{AB}\right|<1.0\) ) 73 . Table 1 indicates a strong positive correlation between SCP and overall LCL. Linear regression analysis in Fig.  4 A demonstrates a significant correlation between SCP and urban LCL ( R 2  = 0.42, p  < 0.001), with notable differences exist among cities, consistent with Hypothesis 1 .

Correlation analysis of SCP and each low-carbon dimension. Pearson correlation analysis effectively measures the strength of linear relationships between two variables, but it does not identify causal relationships between them. To address this limitation and explore the interaction between the two variables, this study sets and solves the closeness coefficient for each low-carbon dimension, which are low-carbon economy (C LCE ), low-carbon society (C LCS ), low-carbon environmental quality (C LCEQ ), low-carbon management (C LCM ), and low-carbon culture (C LCC ). It then calculates the correlation analysis results for SCP and each low-carbon dimension for Hypothesis 1 , as shown in Table 1 . Furthermore, the results of the linear regression analysis are presented in Fig.  4 .

In detail, strong correlations exist between SCP and LCM, LCS, and LCEQ. The correlation is moderate with LCE and weak with LCC. Furthermore, linear regression analysis shows that the links between SCP and low-carbon levels across five dimensions are significant with minimal variance. Cities with higher SCP typically show higher values in LCM ( R 2  = 0.38, p  = 0.000), LCS ( R 2  = 0.35, p  = 0.000), and LCE ( R 2  = 0.32, p  = 0.000) as depicted in Fig.  4 B–D. However, this trend is less pronounced in LCEQ ( R 2  = 0.17, p  = 0.000) and LCC ( R 2  = 0.06, p  = 0.001), which exhibit greater dispersion as shown in Fig.  4 E,F. The lower R 2 values for LCEQ and LCC compared to other dimensions suggest a greater influence of factors not included in the model. Furthermore, to ensure the credibility and reliability of the research findings, this study conducted a sensitivity analysis by identifying and removing outliers from the sample dataset using the Z-score method, in addition to the previously mentioned Pearson correlation analysis. The Pearson correlation coefficient for the original dataset of city SCP and LCL is 0.65, with a significant P-value. After removing the outliers, the Pearson correlation coefficient is 0.61, and the P-value remained significant. Therefore, the correlation between city SCP and LCL proposed in Research Hypothesis 1 is robust.

Coupling coordination degree of SCP and LCL in sample cities

The degree of coupling coordination comprehensively considers multiple aspects of urban complex systems, including economic, social, and environmental dimensions. By systematically evaluating the coordinated development level of urban SCP and LCL, this approach enables the analysis of the coupling and coordination relationships between SCP and LCL, as well as among various subsystems such as LCM, LCS, LCE, LCEQ, and LCC. This reveals the dynamic interactions and causality between SCP and LCL within urban complex systems. The coupling coordination degrees of SCP and LCL, along with their subsystems, in 52 typical smart and low-carbon pilot cities in China, are illustrated in Fig.  5 .

Coupled coordination degree of SCP and LCL, LCS, LCEQ, LCE, LCM, LCC.

Characteristics of objective changes in the coupled coordination degree between SCP and LCL. Based on the coupling coordination model and Eqs. ( 12 ) to ( 14 ), the coupling coordination degree of the urban complex system in SCP and LCL regions is calculated for Hypothesis 1 , as illustrated in Fig.  5 .

From the holistic perspective of urban complex systems, as the level of urban SCP continuously improves, the coupling coordination degree between SCP and LCL among 52 pilot cities in China shows an upward trend. This indicates that as the functional indices of urban SCP and LCL both strengthen, their interaction and coordination also enhance. Among these, Jincheng has the lowest coupled coordination degree at 0.5201, while Beijing boasts the highest at 0.8622. Within the 52 pilot cities, 5.78% exhibit a barely coupling coordination level, 51.93% display a primary coupling coordination level, 25% achieve an intermediate coupling coordination level, and 17.31% reach a good coupling coordination level. Moreover, the average coupling coordination degree of the 52 pilot cities is 0.598, suggesting that the SCP and LCL of the pilot cities can achieve coupled coordinated development.

Characteristics of objective changes in the coupled coordination degree among SCP, LCM, LCS, LCE, LCEQ, and LCC for Hypothesis 1 are illustrated in Fig.  5 .

From the perspective of urban subsystems, the coupling coordination degrees of LCS & SCP, LCE & SCP, and LCM & SCP all exhibit characteristics of steady fluctuations with an upward trend, while the coupling coordination degree of LCC & SCP shows greater volatility in its upward trend. The coupling coordination degree of LCEQ & SCP demonstrates a trend of initially rising and then declining. Furthermore, the average values of the coupling coordination degrees for LCS & SCP, LCE & SCP, LCM & SCP, LCEQ & SCP, and LCC & SCP are 0.478, 0.761, 0.779, 0.710, and 0.485, respectively. Among these, the pilot cities’ subsystems of LCE, LCM, and LCEQ with SCP exhibit an intermediate level of coupling coordination, while the coupling coordination degrees of LCS and LCC with SCP are on the verge of a dysfunctional recession. This indicates that the causal relationships between urban SCP and the subsystems of urban LCM, LCS, LCE, LCEQ, and LCC vary. Overall, Hypothesis 1 holds true both from the perspective of the city's overall system and from the perspective of its various subsystems.

Discussions and implications

Relationship between scp and lcl of different cities.

Considering the evaluation results of the urban SCP and LCL, four grades of the overall points can be classified, namely, excellent (0.7–1.0), average (0.5–0.7), below average (0.4–0.5), and poor (0–0.4). Subsequently, the sample cities in Supplementary Appendix Table A5 were classified based on these gradations. In the sample, cities with excellent SCP constitute 9.62%, about double the proportion with excellent LCL. Cities with average SCP account for 48.08%, whereas those at average LCL represent only 26.92%. Notably, cities with poor LCL comprise 26.92%, nearly triple the rate of those with poor SCP. The findings suggest that China’s SCP currently outperforms its low-carbon city initiatives, largely attributable to the rapid advancement of the Internet and Information and Communication Technology (ICT) in recent years. What’s more, Fig.  4 illustrates that urban SCP significantly positively influences the urban LCL, though substantial variations exist among different cities. The relevant types can be summarized into the following four categories.

Quadrant I-high SCP and high LCL, including only six cities (Shenzhen, Shanghai, Beijing, Ningbo, Xiamen, and Qingdao). These cities are not only among China’s earliest smart city pilots but also recent focus areas for the government’s “Carbon Peak Pioneer Cities” initiative. By actively exploring innovative models, systems, and technologies for smart and low-carbon co-development, these cities provide valuable practical experiences for others. For instance, Shenzhen has developed a multi-level, multi-component greenhouse gas monitoring network and technology system for “carbon flux, carbon concentration, carbon emissions”, while Ningbo has constructed a “smart zero-carbon” comprehensive demonstration port area.

Quadrant II-poor SCP and poor LCL, numerous cities in Fig.  4 A, such as Jincheng, Lhasa, and Urumqi, exhibit poor SCP and LCL. Despite China having the most smart and low-carbon city pilots globally, its development level in these areas still lags significantly behind typical developed countries. While China’s infrastructure like networking and computing power has reached a certain scale, issues persist with insufficient integration and intensity in infrastructure construction and operation, as well as problems with aging infrastructure and low levels of intelligence. Furthermore, although China’s low-carbon pilot cities have made positive progress in promoting low-carbon development, most still have incomplete carbon emission statistical systems and inadequate operational mechanisms, leading to generally poor overall low-carbon development levels.

Quadrant III-high LCL but poor SCP, such as Kunming, Xining, and Guiyang. These cities possess resources conducive to low-carbon development, such as Kunming and Guiyang with their rich forest carbon sinks, and Xining with abundant clean energy sources like solar and wind power. However, they are mostly situated in China’s central and southwest areas with underdeveloped physical and economic conditions. Leveraging their abundant low-carbon resources, and utilizing big data and IoT technology, achieving sustainable green economic growth through carbon credits and trading markets, as well as green finance, represents a significant future development direction for these cities.

Quadrant IV-high SCP but poor LCL, including Suzhou, and Jinhua Zhongshan, decoupling economic development from carbon emissions presents a significant development challenge for these cities. Specifically, for Suzhou, one of the world’s largest industrial cities, the main challenge is achieving decarburization in the energy sector and transitioning high-emission manufacturing industries to low-carbon alternatives.

What’s more, as illustrated in Fig.  5 , the degree of interaction between SCP and LCL across the 52 pilot cities in China positively impacts the balanced and comprehensive performance of these cities. This, in turn, fosters the coordinated development of urban systems as a whole. Moreover, the continual increase in the coupled coordination degree between SCP and LCL with the enhancement of SCP in pilot cities indicates that smart city construction contributes to urban low-carbon development. Future urban development in China should fully leverage the industrial upgrading effect, carbon sequestration effect, and energy utilization effect of smart city construction. However, the increasing slope of the SCP & LCL coupled coordination degree curve in Fig.  5 suggests significant regional differences in the level of SCP & LCL coupled coordination development across Chinese cities. Smart city construction has a more pronounced decarburization effect in central and western cities, southern cities, non-environmentally focused cities, and resource-based cities, with cities in the northwest showing notably poorer levels of SCP & LCL coupled coordination development. This serves as a warning for future urban development in China.

Relationships between SCP and LCL in each urban subsystem

The relationship between urban SCP and LCL across five dimensions is illustrated in Fig.  4 B–F. There is a strong positive correlation between SCP and LCM, LCS, and LCE, while a moderate correlation is observed with LCEQ, and a weak correlation with LCC. Furthermore, the degree of coupling coordination between SCP and subsystems such as LCS, LCEQ, LCE, LCM, and LCC is examined in Fig.  5 . The results of the coupling coordination vividly illustrate the synergistic interactions and developmental harmony between urban SCP and various systems.

Among these, the coupling coordination degree curve fluctuation between SCP & LCM is stable, situated at an intermediate coupling coordination level, indicating the dominant role of the Chinese government in the construction of smart cities and low-carbon cities, as well as the effectiveness of policy implementation. However, this also suggests that in promoting urban smart and low-carbon construction, China faces the risk of adopting “one-size-fits-all” mandatory policies, neglecting to advance construction in phases with emphasis, tailored to the city's resource endowment and economic development status. The coupling coordination degree curve changes between SCP&LCE and SCP&LCL show the highest degree of fit, indicating that low-carbon economic development brought about by digital empowerment and upgrading of the urban industrial structure is a key driving factor for promoting the coupled coordination development of urban smart and low-carbon initiatives. Transforming traditional industrial structures and pursuing low-carbon upgrades of the economic structure present challenges for urban development in China today. The coupled coordination degree of SCP & LCS is on the verge of a dysfunctional recession, highlighting the imbalance in the development between China's SCP and LCS, especially in terms of new infrastructure construction, such as smart transportation and logistics facilities, smart energy systems, smart environmental resources facilities, etc. The current construction of new infrastructure in China is far from meeting the living needs of the broad masses of people.

It is noteworthy that with the continuous improvement of the SCP in sample cities, the coupling performance degree between SCP and LCEQ exhibits two phases: an initial stage of synergistic enhancement followed by a stage of diminished synergy. In the early phase of synergistic development, the SCP and LCEQ systems of cities, driven by shared goals of sustainable urban development, strategy adjustments, resource sharing, and technological progress, facilitated effective collaboration and integration between systems. However, upon reaching a certain stage, intensified resource competition, declining management efficiency, and environmental changes led to internal system fatigue, resulting in weakened synergy. This indicates that once the technological effects generated by smart city construction reach a certain level, it becomes crucial to enhance the city's capacity for autonomous innovation. Addressing the bottleneck issues of core technologies and transforming the development mode of smart low-carbon technology from “imitative innovation” represent significant breakthroughs for further promoting the coupled coordination of SCP and LCEQ in China’s future.

Moreover, as the SCP of sample cities continuously improves, the coupled coordination degree between SCP and LCC shows two phases: initial stable fluctuations and subsequent rapid growth. The turning point in the curve change occurs at a coupled coordination degree of 0.6, denoted as the primary coupling coordination point. Among these, the low-carbon awareness rate of urban residents, as a key indicator of LCC, shows that the majority of urban residents in China are still in the cognitive awakening stage regarding low-carbon consciousness. At this stage, residents begin to recognize the severity of climate change and environmental degradation, along with the importance of smart low-carbon lifestyles in mitigating these issues. The government continuously promotes this awareness through media reports, educational activities, official propaganda, and community initiatives. As residents gain a deeper understanding of the issues, their attitudes shift from initial indifference or skepticism to a stronger identification with and support for the values and concepts of smart low-carbon living. This shift encourages residents to experiment with new smart low-carbon lifestyles, gradually finding suitable smart low-carbon behavioral patterns that become habitual. Ultimately, when smart low-carbon lifestyles are fully internalized as part of residents’ values, they not only practice smart low-carbon living at the individual level but also actively participate in promoting society’s smart low-carbon construction. Therefore, this study posits that the emergence of the coupled coordination degree turning point between SCP and LCC is not only a process of individual behavioral change but also a reflection of social and cultural transformation. This process is time-consuming and influenced by multiple factors, including policy guidance, economic incentives, educational dissemination, and the social atmosphere.

Implications for promoting coupling coordination development between urban SCP and LCL

Low-carbon and smartness are vital features of modern, sustainable urban development and key supports for it. This study posits that urban low-carbon and smart development should not be disjointed but rather synergistic and complementary. To better achieve sustainable urban development goals, a model should be constructed with “low-carbon” as the cornerstone of sustainable development and “smartness” as the technological assurance for low-carbon growth. Specifically, this study proposes the “urban smart low-carbon co-development model”, which entails a deep integration of intelligent technologies such as the Internet of Things (IoT) and big data with urban construction, governance services, and economic development. This model leverages digitalization to facilitate decarburization, thereby achieving urban sustainable development goals such as energy-efficient and green urbanization, ecological and livable environments, and streamlined governance services.

Furthermore, to better coordinate smart development with low-carbon city construction, enhance low-carbon city building through digitalization, and explore exemplary practices and models of smart low-carbon city construction, this study finds it necessary to establish an evaluation system for smart and low-carbon urban co-development. Therefore, based on the aforementioned urban SCP and LCL evaluation indicator system, this study initially conducted a literature review of past research, selecting 5 primary indicators and 20 secondary indicators from 48 articles to evaluate the degree of coupling coordination development between urban SCP and LCL. Subsequently, the Delphi method was employed to finalize the list of evaluation indicators, with 10 experts from various regions and diverse backgrounds in China refining the list and determining the weights of each indicator, as shown in Supplementary Appendix Table A6 . The final Smart Low-Carbon City Coupling Coordination Development Evaluation Indicator System, as presented in Table 3 , comprises 5 primary indicators and 18 secondary indicators. This evaluation system aims to emphasize the utilization of next-generation information technologies such as 5G, artificial intelligence, cloud computing, and blockchain to expand urban green ecological spaces, strengthen ecological environment governance, and enhance the level of intelligent urban governance, meeting the development needs of smart low-carbon cities.

The policy implications from the analysis results suggest that actions should be taken by government departments in China to reduce the uneven performance between urban SCP and LCL across various cities. These actions include, for example: Firstly, guiding the innovative development of urban SCP and LCL through policies, such as enhancing government digital services and administrative platforms, continuously promoting the development of emerging industries and the upgrading of traditional industries, and actively promoting green energy technologies. Secondly, categorizing and advancing the coordinated development of smart and low-carbon cities—comprehensive development should be pursued simultaneously in large cities in eastern and central China, while in smaller cities in western China, priorities should include enhancing urban innovation capabilities and improving infrastructure to lay a solid foundation for the coupled coordination of urban SCP and LCL. Thirdly, constructing a multi-stakeholder governance system to maximize the leading role of the government, the main role of enterprises, and the active participation of residents. By fostering a positive social atmosphere and cultural attributes, this will enhance the sense of participation and achievement among different social groups, creating a sustainable development model for urban SCP and LCL coordination. Lastly, emphasizing the development of SCP and LCL coordination in county-level cities is crucial. While large Chinese cities have already begun to form a pattern of coordinated SCP and LCL development, county-level cities, though with weaker infrastructures, possess tremendous potential. Focusing on low-carbon production, circulation, and consumption, and strengthening smart and low-carbon constructions in county-level cities will be a vital task for future urban development in China.

Conclusions

The global urbanization process brings opportunities for economic growth and social development, but also presents a series of challenges, such as environmental pressures and resource constraints 3 . The evaluation of urban SCP and LCL creates a link between the policy-making in urban resources environment management and the objectives of sustainable development goals (SDGs 11.4, 11.6, and 11.b) at the city level 74 . Currently, there is no unified consensus on the coupling coordination development between urban SCP and LCL. This study proposes a method combining qualitative and quantitative analysis from the perspective of urban complex systems to analyze the coupling coordination relationship between SCP and LCL. This new method clearly interprets a strong positive correlation between urban smart performance and the overall low-carbon level. Specifically, there are strong correlations between SMC and LCM, LCS, and LCE, with a moderate correlation to LCQE and a weak correlation with LCC. Several innovative insights for this method are highlighted: (i) sustainable development based on SCP and LCL assessment; (ii) emphasizing the “people-centric” concept in urban development; (iii) analyzing from the perspective of urban complex systems.

This study selected 52 typical smart and low-carbon pilot cities in China as sample cities to analyze the coupled coordination relationship between urban SCP and LCL. And the main findings from this analysis can be summarized as follows: (i) smart city initiatives outperform low-carbon city development, with notable differences in SCP and LCL effectiveness across eastern, central, and non-resource-based cities versus western, peripheral, and resource-dependent ones in China. (ii) A strong positive link between urban SCP and low-carbon levels, especially between SCP and LCM, LCS, and LCE, with moderate and weak correlations to LCEQ and LCC, respectively. (iii) An increasing urban SCP levels enhance the coupling coordination within the urban SCP and LCL system. SCP & LCE, SCP & LCM, and SCP & LCS subsystems align well with the overall system, driving the coupled coordination of urban SCP and LCL. In contrast, SCP & LCC and SCP & LCEQ have lesser alignment, affected by factors like technology, policy, economic incentives, education, and societal attitudes. Based on the evaluation results, this study posits that the development of urban low-carbon and smart initiatives should not be disjointed but rather synergistic and complementary. This study constructs an evaluation indicator system for the co-development of smart low-carbon cities aimed at better guiding the future coupling coordination development of smart and low-carbon cities.

The novelty of this study not only addresses the practical dilemma of obtaining comprehensive, accurate, and timely urban-level carbon emission data, a challenge due to existing measurement and estimation technologies being unable to capture all types of carbon emissions, but also assesses the urban SCP and LCL. Simultaneously, by combining qualitative and quantitative analysis methods, it fills the research gap on the nature of the coupled coordination relationship between urban SCP and LCL. Moreover, from the perspective of urban complex systems, this study dissects the urban low-carbon level into LCC, LC, LCE, LCEQ, and LCS, exploring their respective coupled coordination relationships with SCP. This clarifies the impact mechanism between SCP and LCL, providing a theoretical basis for smart low-carbon city co-development. The limitations of the study are also appreciated. Firstly, the study only selected a sample of cities in China, and the limited number of samples may not fully substantiate the research conclusions. Secondly, the indicator system constructed by this study is still not perfect, leading to certain inaccuracies in the evaluation results. In this regard, future studies are recommended to conduct a more comprehensive comparison analysis on the coupled coordination relationship between SCP and LCL at city, regional, and national levels, which would be beneficial in better guiding the practice of urban sustainability.

Data availability

All data generated or analysed during this study are included in this published article [and its Supplementary Information files].

Zheng, H. W., Shen, G. Q. & Wang, H. A review of recent studies on sustainable urban renewal. Habit. Int. 41 , 272–279. https://doi.org/10.1016/j.habitatint.2013.08.006 (2014).

Article   CAS   Google Scholar  

Bibri, S. E. & Krogstie, J. Smart sustainable cities of the future: An extensive interdisciplinary literature review. Sustain. Cities Soc. 31 , 183–212. https://doi.org/10.1016/j.scs.2017.02.016 (2017).

Article   Google Scholar  

Chen, M., Liu, W. & Lu, D. Challenges and the way forward in China’s new-type urbanization. Land Use Policy 55 , 334–339. https://doi.org/10.1016/j.landusepol.2015.07.025 (2016).

Liang, W. & Yang, M. Urbanization, economic growth and environmental pollution: Evidence from China. Sustain. Comput. Inform. Syst. 21 , 1–9. https://doi.org/10.1016/j.suscom.2018.11.007 (2019).

Guan, X., Wei, H., Lu, S., Dai, Q. & Su, H. Assessment on the urbanization strategy in China: Achievements, challenges and reflections. Habit. Int. 71 , 97–109. https://doi.org/10.1016/j.habitatint.2017.11.009 (2018).

Wu, H., Hao, Y. & Weng, J.-H. How does energy consumption affect China’s urbanization? New evidence from dynamic threshold panel models. Energy Policy 127 , 24–38. https://doi.org/10.1016/j.enpol.2018.11.057 (2019).

Liu, H., Cui, W. & Zhang, M. Exploring the causal relationship between urbanization and air pollution: Evidence from China. Sustain. Cities Soc. 80 , 783. https://doi.org/10.1016/j.scs.2022.103783 (2022).

Tang, F. et al. Spatio-temporal variation and coupling coordination relationship between urbanisation and habitat quality in the Grand Canal, China. Land Use Policy 117 , 6119. https://doi.org/10.1016/j.landusepol.2022.106119 (2022).

Kim, J. Smart city trends: A focus on 5 countries and 15 companies. Cities 123 , 551. https://doi.org/10.1016/j.cities.2021.103551 (2022).

Silva, B. N., Khan, M. & Han, K. Towards sustainable smart cities: A review of trends, architectures, components, and open challenges in smart cities. Sustain. Cities Soc. 38 , 697–713. https://doi.org/10.1016/j.scs.2018.01.053 (2018).

Yigitcanlar, T., Kankanamge, N. & Vella, K. How are smart city concepts and technologies perceived and utilized? A systematic geo-twitter analysis of smart cities in Australia. J. Urban Technol. 28 , 135–154. https://doi.org/10.1080/10630732.2020.1753483 (2021).

Yigitcanlar, T. et al. Can cities become smart without being sustainable? A systematic review of the literature. Sustain. Cities Soc. 45 , 348–365. https://doi.org/10.1016/j.scs.2018.11.033 (2019).

Guo, Q. & Zhong, J. The effect of urban innovation performance of smart city construction policies: Evaluate by using a multiple period difference-in-differences model. Technol. Forecast. Soc. Change 184 , 2003. https://doi.org/10.1016/j.techfore.2022.122003 (2022).

Ismagilova, E., Hughes, L., Dwivedi, Y. K. & Raman, K. R. Smart cities: Advances in research—An information systems perspective. Int. J. Inf. Manag. 47 , 88–100. https://doi.org/10.1016/j.ijinfomgt.2019.01.004 (2019).

Caragliu, A. & Del Bo, C. F. Smart innovative cities: The impact of Smart City policies on urban innovation. Technol. Forecast. Soc. Change 142 , 373–383. https://doi.org/10.1016/j.techfore.2018.07.022 (2019).

Yigitcanlar, T. et al. Understanding ‘smart cities’: Intertwining development drivers with desired outcomes in a multidimensional framework. Cities 81 , 145–160. https://doi.org/10.1016/j.cities.2018.04.003 (2018).

Liu, Z. et al. Decision optimization of low-carbon dual-channel supply chain of auto parts based on smart city architecture. Complexity 2020 , 5951. https://doi.org/10.1155/2020/2145951 (2020).

Guo, Q., Wang, Y. & Dong, X. Effects of smart city construction on energy saving and CO 2 emission reduction: Evidence from China. Appl. Energy 313 , 879. https://doi.org/10.1016/j.apenergy.2022.118879 (2022).

Cheng, J., Yi, J., Dai, S. & Xiong, Y. Can low-carbon city construction facilitate green growth? Evidence from China’s pilot low-carbon city initiative. J. Clean. Prod. 231 , 1158–1170. https://doi.org/10.1016/j.jclepro.2019.05.327 (2019).

Sun, W. & Huang, C. Predictions of carbon emission intensity based on factor analysis and an improved extreme learning machine from the perspective of carbon emission efficiency. J. Clean. Prod. 338 , 414. https://doi.org/10.1016/j.jclepro.2022.130414 (2022).

Shi, B., Li, N., Gao, Q. & Li, G. Market incentives, carbon quota allocation and carbon emission reduction: Evidence from China’s carbon trading pilot policy. J. Environ. Manag. 319 , 650. https://doi.org/10.1016/j.jenvman.2022.115650 (2022).

Sun, L. et al. Carbon emission transfer strategies in supply chain with lag time of emission reduction technologies and low-carbon preference of consumers. J. Clean. Prod. 264 , 664. https://doi.org/10.1016/j.jclepro.2020.121664 (2020).

Matsumura, E. M., Prakash, R. & Vera-Munoz, S. C. Firm-value effects of carbon emissions and carbon disclosures. Acc. Rev. 89 , 695–724. https://doi.org/10.2308/accr-50629 (2014).

Lv, M. & Bai, M. Evaluation of China’s carbon emission trading policy from corporate innovation. Financ. Res. Lett. 39 , 565. https://doi.org/10.1016/j.frl.2020.101565 (2021).

Jia, Z. & Lin, B. Rethinking the choice of carbon tax and carbon trading in China. Technol. Forecast. Soc. Change 159 , 187. https://doi.org/10.1016/j.techfore.2020.120187 (2020).

Huo, T., Xu, L., Liu, B., Cai, W. & Feng, W. China’s commercial building carbon emissions toward 2060: An integrated dynamic emission assessment model. Appl. Energy 325 , 828. https://doi.org/10.1016/j.apenergy.2022.119828 (2022).

Lin, B. & Huang, C. Analysis of emission reduction effects of carbon trading: Market mechanism or government intervention? Sustain. Prod. Consump. 33 , 28–37. https://doi.org/10.1016/j.spc.2022.06.016 (2022).

Zhang, M. & Liu, Y. Influence of digital finance and green technology innovation on China’s carbon emission efficiency: Empirical analysis based on spatial metrology. Sci. Total Environ. 838 , 463. https://doi.org/10.1016/j.scitotenv.2022.156463 (2022).

Zhu, X. & Li, D. How to promote the construction of low-carbon cities in China? An urban complex ecosystem perspective. Sustain. Dev. https://doi.org/10.1002/sd.2897 (2024).

He, C., Zhang, D., Huang, Q. & Zhao, Y. Assessing the potential impacts of urban expansion on regional carbon storage by linking the LUSD-urban and InVEST models. Environ. Model. Softw. 75 , 44–58. https://doi.org/10.1016/j.envsoft.2015.09.015 (2016).

Nowak, D. J., Greenfield, E. J., Hoehn, R. E. & Lapoint, E. Carbon storage and sequestration by trees in urban and community areas of the United States. Environ. Pollut. 178 , 229–236. https://doi.org/10.1016/j.envpol.2013.03.019 (2013).

Article   CAS   PubMed   Google Scholar  

Wang, T. et al. Mobility based trust evaluation for heterogeneous electric vehicles network in smart cities. IEEE Trans. Intell. Transp. Syst. 22 , 1797–1806. https://doi.org/10.1109/tits.2020.2997377 (2021).

Huovila, A., Bosch, P. & Airaksinen, M. Comparative analysis of standardized indicators for Smart sustainable cities: What indicators and standards to use and when? Cities 89 , 141–153. https://doi.org/10.1016/j.cities.2019.01.029 (2019).

Nizetic, S., Djilali, N., Papadopoulos, A. & Rodrigues, J. J. P. C. Smart technologies for promotion of energy efficiency, utilization of sustainable resources and waste management. J. Clean. Prod. 231 , 565–591. https://doi.org/10.1016/j.jclepro.2019.04.397 (2019).

Sharifi, S., Saman, W. & Alemu, A. Identification of overheating in the top floors of energy-efficient multilevel dwellings. Energy Build. https://doi.org/10.1016/j.enbuild.2019.109452 (2019).

Shafiq, M., Tian, Z., Sun, Y., Du, X. & Guizani, M. Selection of effective machine learning algorithm and Bot–IoT attacks traffic identification for internet of things in smart city. Future Gener. Comput. Syst. Int. J. Esci. 107 , 433–442. https://doi.org/10.1016/j.future.2020.02.017 (2020).

Huang, S., Liu, A., Zhang, S., Wang, T. & Xiong, N. N. BD-VTE: A novel baseline data based verifiable trust evaluation scheme for smart network systems. IEEE Trans. Netw. Sci. Eng. 8 , 2087–2105. https://doi.org/10.1109/tnse.2020.3014455 (2021).

Reed, M. S. et al. Evaluating impact from research: A methodological framework. Res. Policy 50 , 147. https://doi.org/10.1016/j.respol.2020.104147 (2021).

Venable, J., Pries-Heje, J. & Baskerville, R. FEDS: A framework for evaluation in design science research. Eur. J. Inf. Syst. 25 , 77–89. https://doi.org/10.1057/ejis.2014.36 (2016).

Kristan, M. et al. A novel performance evaluation methodology for single-target trackers. IEEE Trans. Pattern Anal. Mach. Intell. 38 , 2137–2155. https://doi.org/10.1109/tpami.2016.2516982 (2016).

Article   PubMed   Google Scholar  

Li, H. Research progress on evaluation methods and factors influencing shale brittleness: A review. Energy Rep. 8 , 4344–4358. https://doi.org/10.1016/j.egyr.2022.03.120 (2022).

Lyu, H.-M., Zhou, W.-H., Shen, S.-L. & Zhou, A.-N. Inundation risk assessment of metro system using AHP and TFN-AHP in Shenzhen. Sustain. Cities Soc. 56 , 103. https://doi.org/10.1016/j.scs.2020.102103 (2020).

Buyukozkan, G. & Guleryuz, S. An integrated DEMATEL-ANP approach for renewable energy resources selection in Turkey. Int. J. Prod. Econ. 182 , 435–448. https://doi.org/10.1016/j.ijpe.2016.09.015 (2016).

Ervural, B. C., Zaim, S., Demirel, O. F., Aydin, Z. & Delen, D. An ANP and fuzzy TOPSIS-based SWOT analysis for Turkey’s energy planning. Renew. Sustain. Energy Rev. 82 , 1538–1550. https://doi.org/10.1016/j.rser.2017.06.095 (2018).

Gao, Z. et al. EEG-based spatio-temporal convolutional neural network for driver fatigue evaluation. IEEE Trans. Neural Netw. Learn. Syst. 30 , 2755–2763. https://doi.org/10.1109/tnnls.2018.2886414 (2019).

Manzano, A. The craft of interviewing in realist evaluation. Evaluation 22 , 342–360. https://doi.org/10.1177/1356389016638615 (2016).

Zeng, S., Jin, G., Tan, K. & Liu, X. Can low-carbon city construction reduce carbon intensity? Empirical evidence from low-carbon city pilot policy in China. J. Environ. Manag. 332 , 363. https://doi.org/10.1016/j.jenvman.2023.117363 (2023).

Liu, X., Li, Y., Chen, X. & Liu, J. Evaluation of low carbon city pilot policy effect on carbon abatement in China: An empirical evidence based on time-varying DID model. Cities 123 , 582. https://doi.org/10.1016/j.cities.2022.103582 (2022).

Tan, S. et al. A holistic low carbon city indicator framework for sustainable development. Appl. Energy 185 , 1919–1930. https://doi.org/10.1016/j.apenergy.2016.03.041 (2017).

Article   ADS   Google Scholar  

Shi, X. & Xu, Y. Evaluation of China’s pilot low-carbon city program: A perspective of industrial carbon emission efficiency. Atmos. Pollut. Res. 13 , 446. https://doi.org/10.1016/j.apr.2022.101446 (2022).

Yang, S., Pan, Y. & Zeng, S. Decision making framework based Fermatean fuzzy integrated weighted distance and TOPSIS for green low-carbon port evaluation. Eng. Appl. Artif. Intell. 114 , 5048. https://doi.org/10.1016/j.engappai.2022.105048 (2022).

Fang, G., Gao, Z., Tian, L. & Fu, M. What drives urban carbon emission efficiency?—Spatial analysis based on nighttime light data. Appl. Energy 312 , 772. https://doi.org/10.1016/j.apenergy.2022.118772 (2022).

Yang, S., Jahanger, A. & Hossain, M. R. How effective has the low-carbon city pilot policy been as an environmental intervention in curbing pollution? Evidence from Chinese industrial enterprises. Energy Econ. 118 , 523. https://doi.org/10.1016/j.eneco.2023.106523 (2023).

Huang, G., Li, D., Zhu, X. & Zhu, J. Influencing factors and their influencing mechanisms on urban resilience in China. Sustain. Cities Soc. 74 , 210. https://doi.org/10.1016/j.scs.2021.103210 (2021).

Li, W. et al. Carbon emission and economic development trade-offs for optimizing land-use allocation in the Yangtze River Delta, China. Ecol. Indic. 147 , 950. https://doi.org/10.1016/j.ecolind.2023.109950 (2023).

Wu, H. et al. Exploring the impact of urban form on urban land use efficiency under low-carbon emission constraints: A case study in China’s Yellow River Basin. J. Environ. Manag. 311 , 866. https://doi.org/10.1016/j.jenvman.2022.114866 (2022).

Zhao, S. et al. Has China’s low-carbon strategy pushed forward the digital transformation of manufacturing enterprises? Evidence from the low-carbon city pilot policy. Environ. Impact Assess. Rev. 102 , 184. https://doi.org/10.1016/j.eiar.2023.107184 (2023).

Pan, A., Zhang, W., Shi, X. & Dai, L. Climate policy and low-carbon innovation: Evidence from low-carbon city pilots in China. Energy Econ. 112 , 129. https://doi.org/10.1016/j.eneco.2022.106129 (2022).

De Jong, M., Joss, S., Schraven, D., Zhan, C. & Weijnen, M. Sustainable-smart-resilient-low carbon-eco-knowledge cities; making sense of a multitude of concepts promoting sustainable urbanization. J. Clean. Prod. 109 , 25–38. https://doi.org/10.1016/j.jclepro.2015.02.004 (2015).

He, B.-J. et al. Co-benefits approach: Opportunities for implementing sponge city and urban heat island mitigation. Land Use Policy 86 , 147–157. https://doi.org/10.1016/j.landusepol.2019.05.003 (2019).

Nizetic, S., Solic, P., Lopez-de-Ipina, D. & Patrono, L. Internet of Things (IoT): Opportunities, issues and challenges towards a smart and sustainable future. J. Clean. Prod. 274 , 877. https://doi.org/10.1016/j.jclepro.2020.122877 (2020).

Abduljabbar, R. L., Liyanage, S. & Dia, H. The role of micro-mobility in shaping sustainable cities: A systematic literature review. Transp. Res. D Transp. Environ. 92 , 734. https://doi.org/10.1016/j.trd.2021.102734 (2021).

Anh Tuan, H., Van Viet, P. & Xuan Phuong, N. Integrating renewable sources into energy system for smart city as a sagacious strategy towards clean and sustainable process. J. Clean. Prod. 305 , 7161. https://doi.org/10.1016/j.jclepro.2021.127161 (2021).

March, H. & Ribera-Fumaz, R. Smart contradictions: The politics of making Barcelona a self-sufficient city. Eur. Urban Reg. Stud. 23 , 816–830. https://doi.org/10.1177/0969776414554488 (2016).

Yigitcanlar, T. & Lee, S. H. Korean ubiquitous-eco-city: A smart-sustainable urban form or a branding hoax? Technol. Forecast. Soc. Change 89 , 100–114. https://doi.org/10.1016/j.techfore.2013.08.034 (2014).

Kumar, S., Sharma, D., Rao, S., Lim, W. M. & Mangla, S. K. Past, present, and future of sustainable finance: Insights from big data analytics through machine learning of scholarly research. Ann. Oper. Res. https://doi.org/10.1007/s10479-021-04410-8 (2022).

Article   PubMed   PubMed Central   Google Scholar  

Li, Y., Gao, P., Tang, B., Yi, Y. & Zhang, J. Double feature extraction method of ship-radiated noise signal based on slope entropy and permutation entropy. Entropy 24 , 22. https://doi.org/10.3390/e24010022 (2022).

Article   ADS   CAS   Google Scholar  

Zavadskas, E. K., Mardani, A., Turskis, Z., Jusoh, A. & Nor, K. M. D. Development of TOPSIS method to solve complicated decision-making problems: An overview on developments from 2000 to 2015. Int. J. Inf. Technol. Decis. Making 15 , 645–682. https://doi.org/10.1142/s0219622016300019 (2016).

Edelmann, D., Mori, T. F. & Szekely, G. J. On relationships between the Pearson and the distance correlation coefficients. Stat. Probab. Lett. 169 , 960. https://doi.org/10.1016/j.spl.2020.108960 (2021).

Article   MathSciNet   Google Scholar  

Wang, S., Kong, W., Ren, L., Zhi, D. & Dai, B. Research on misuses and modification of coupling coordination degree model in China. J. Nat. Resour. 36 , 793–810 (2021).

CAS   Google Scholar  

Baak, M., Koopman, R., Snoek, H. & Klous, S. A new correlation coefficient between categorical, ordinal and interval variables with Pearson characteristics. Comput. Stat. Data Anal. 152 , 7043. https://doi.org/10.1016/j.csda.2020.107043 (2020).

Saadatmorad, M., Talookolaei, R.-A.J., Pashaei, M.-H., Khatir, S. & Wahab, M. A. Pearson correlation and discrete wavelet transform for crack identification in steel beams. Mathematics 10 , 689. https://doi.org/10.3390/math10152689 (2022).

De Winter, J. C. F., Gosling, S. D. & Potter, J. Comparing the Pearson and Spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychol. Methods 21 , 273–290. https://doi.org/10.1037/met0000079 (2016).

Belz, F. M. & Binder, J. K. Sustainable entrepreneurship: A convergent process model. Bus. Strategy Environ. 26 , 1–17. https://doi.org/10.1002/bse.1887 (2017).

Download references

The funding provided by Government of Jiangsu Province (BE2022606 and BM2022035).

Author information

Authors and affiliations.

Department of Construction and Real Estate, School of Civil Engineering, Southeast University, Nanjing, 210018, China

Xiongwei Zhu, Dezhi Li, Shenghua Zhou & Lugang Yu

Engineering Research Center of Building Equipment, Energy, and Environment, Ministry of Education, Southeast University, Nanjing, 210018, China

Department of Wood Science, The University of British Columbia, Vancouver, V6T 1Z4, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

X.Z: conceptualization, methodology, formal analysis, investigation, writing-original draft. D. L: supervision, project administration, funding acquisition. S. Z: writing-review & editing. S.Z: writing-review & editing. L.Y: data curation.

Corresponding author

Correspondence to Dezhi Li .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary tables., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Zhu, X., Li, D., Zhou, S. et al. Evaluating coupling coordination between urban smart performance and low-carbon level in China’s pilot cities with mixed methods. Sci Rep 14 , 20461 (2024). https://doi.org/10.1038/s41598-024-68417-4

Download citation

Received : 28 March 2024

Accepted : 23 July 2024

Published : 03 September 2024

DOI : https://doi.org/10.1038/s41598-024-68417-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Smart city performance (SCP)
• Low-carbon level (LCL)
• ETPC analysis model
• Coupling coordination degree
• Smart low-carbon coupling coordination development paths

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.

• Open access
• Published: 05 September 2024

Whether medicine supply is really meeting primary health care needs: a mixed-methods study in Shandong Province, China

• Zhixin Fan 1 , 2   na1 ,
• Tiantian Gao 1 , 2 , 3   na1 ,
• Qiang Sun 1 , 2 &
• Zaheer-Ud-Din Babar 4  

Global Health Research and Policy volume  9 , Article number:  32 ( 2024 ) Cite this article

Metrics details

With the aging population, the increasing prevalence of chronic non-communicable diseases, and the diversified needs for primary health care (PHC) medicines, it is necessary to rethink the functional role of the supply of PHC medicines. This study aims to evaluate the supply of PHC medicines and the status of meeting PHC medicine needs.

The mixed-methods study was conducted to evaluate the supply of PHC medicines in Shandong Province. In the quantitative study, survey questionnaires were distributed to county hospitals, township hospitals, and patients, and a prescription review was performed in township hospitals. In the qualitative study, semi-structured interviews were conducted with the pharmacy managers, physicians, and patients in county hospitals, township hospitals, and village clinics. A senior pharmacist from a tertiary hospital who has rich experience on the indications for medicine use, accompanied us on a visit to inspect the PHC pharmacies to survey medicine equipment with a professional perspective.

Quantitative analysis revealed that 211 county hospitals and 1,581 township hospitals participated in the survey, revealing the median annual frequency of medicine shortages of 5.0 times for county hospitals and 2.0 times for township hospitals. Of the 6,323 patient medication surveys, after excluding 152 patients not involved in medication use, 945 (15.3%) indicated medicine shortages, with half of these attributable to institutions lacking required medicines (52.8%). On average, the prescription qualified rate of 37 township hospitals was 72.2%. Four final themes emerged during the qualitative data analysis: (1) Supply of PHC medicines; (2) Solutions to the shortage of off-list medicines; (3) Appropriateness of PHC medicines list; (4) Pharmacist workforce development and pharmacy services.

Conclusions

The discrepancy between patients’ need for PHC medicine and present medicine supply is noteworthy. It is suggested that governments should optimize the existing lists to adequately meet patient medicine needs and prioritize medicines for chronic diseases, which is also particularly important for developing countries. Integrated health care may be a novel strategy to establish unified medicines list and achieve uniform pharmaceutical services in PHC.

Introduction

The Sustainable Development Goals (SDGs) by 2030 focus on achieving universal health coverage (UHC), and an important element of UHC is ensuring access to safe, effective, quality and affordable essential medicines [ 1 ]. Primary health care (PHC) is widely recognized as the most inclusive, equitable, and cost-effective way of achieving UHC [ 2 ]. Integrated health services (IHS), as one of its key components, play an important role in meeting people's health needs throughout their life cycle, including the provision of essential medicines [ 2 ].

Countries around the world have made great efforts to ensure access to essential medicines and strive to achieve UHC. For example, Indonesia has introduced the Jaminan Kesehatan Nasional scheme to provide free access to a list of essential medicines for all citizens [ 3 ]. Brazil’s Unified Health System comprises the Basic Component of Pharmaceutical Services to ensure adequate availability of medicines, which has stated that cities could create lists of medicines that meet local needs [ 4 ]. South Africa has implemented the Central Chronic Medicine Dispensing and Distribution program to improve access to chronic medicines [ 5 ].

To improve access to PHC medicines and achieve UHC, China has implemented a series of major reforms. The National Essential Medicines Policy issued in 2009 aims to improve equitable access to essential medicines [ 6 ]. The government encourages PHC institutions to stock at least 90% of essential medicines and promotes a “1 + X” system (“1” for essential medicines, “X” for non-essential medicines) to optimize and standardize the structure of medicines [ 7 ]. The zero mark-up policy, nationa l volume-based procurement policy (NVBP), and dual invoicing policy have been extended to all public hospitals nationwide to regulate medicine prices and standardize medicine supply [ 8 , 9 , 10 , 11 ]. These policies have led to a reduction in medicine prices and a further improvement in medicine affordability, benefiting PHC similarly [ 12 , 13 ].

In addition, under the establishment of hierarchical diagnosis and treatment, China has innovatively applied IHS to optimize and enhance the supply and access to PHC medicines [ 14 ]. County Medical Communities (CMCs) represent an effective model for integrating and optimizing health service resources to improve PHC. The CMCs model integrates county hospitals, township hospitals, and village clinics into a coordinated, three-level county medical service system [ 15 ]. In 2019, China intensified its efforts to establish CMCs, emphasizing medicine supply as a critical aspect [ 16 ]. Within the CMCs, a county hospital serves as the lead unit, encompassing numerous township hospitals and village clinics as members. The lead unit is responsible for the supply and management of medicines within the CMCs. The government encourages the lead unit to formulate a unified medicine list within the CMCs based on the needs of the service area, from which member units procure their medicines. Moreover, the lead unit regularly reviews and consolidates the medicine procurement plans of member units, organizes unified medicine distribution, and oversees unified medicine procurement to enhance the PHC medicine availability. It is also encouraged to organize regular training for pharmacists from member units to promote standardized pharmacy services [ 17 ]. These policy initiatives have led to an improvement in the supply of PHC medicines and further promoted the accessibility and affordability of medicines [ 18 , 19 , 20 ].

Medicine supply and need are often linked and influence each other. Nowadays, with an aging population and the increasing prevalence of chronic non-communicable diseases (NCDs) [ 21 , 22 ], China has become the country with the largest elderly population and one of the fastest aging countries in the world. By the end of 2023, there will be nearly 297 million people aged 60 and over, accounting for 21.1% of the total population, with 217 million will be aged 65 and over, accounting for 15.4% of the total population [ 23 ]. The high incidence of chronic diseases is another major challenge, with more than 78% of older people suffering from at least one chronic diseases [ 24 ]. PHC institutions are gradually becoming the mainstay to treat elderly and chronic patients, leading to a diversification in the need for PHC medicines. Despite gains in China's PHC capabilities, challenges remain, including a lack of PHC workforce, inadequate capacity, and outdated management practices [ 25 ]. The ideal role of PHC is to treat chronic, common, and frequently occurring diseases [ 2 ]. A large number of high-quality and low-priced medicines have been introduced into the NVBP list [ 26 , 27 ], providing more choices for PHC medicine supply.

In China, medical institutions are in a stage of self-selection and self-management of medicines. Relevant policies stipulate that medical institutions should establish their own basic medicine supply list and optimize the list based on safe, effective, and economical use of medication, as well as the characteristics of the diseases treated in the institutions [ 28 ]. Despite these measures, there is still a need to assess whether the PHC medicine supply is up to date and meets the needs of residents. This requires a comprehensive evaluation involving multiple stakeholders, including medical institutions, physicians, pharmacists, and patients. To ensure a thorough understanding, we conducted a mixed-methods study to evaluate the supply of PHC medicines and the status of meeting PHC medicine needs in Shandong Province, China.

Study design and setting

This mixed-methods study was conducted from July 1, 2023 to January 31, 2024 in Shandong Province, China. Shandong Province, in eastern China, had a total population of 102 million by 2022, making it the second most populous province in China, with 16.7 per cent of the total population aged 65 and above. Additionally, over 50% of older people suffered from at least two chronic diseases [ 29 , 30 ]. The rural population of this province shares many similarities with other rural areas of China in terms of income levels, literacy, and health.

We adopted a sequential explanatory design strategy to first understand the basic situation of the supply of PHC medicines in Shandong Province through quantitative research, and then further in-depth analysis and explanation through qualitative research. The quantitative study consisted of three parts: Part A. a structured questionnaire for all CMCs in Shandong Province; Part B. a medication questionnaire for PHC patients; and Part C. a prescription review for township hospitals. In the qualitative study, semi-structured qualitative interviews were conducted with the pharmacy managers and physicians in county and township hospitals, village doctors in village clinics, and PHC patients. In addition, a professional clinical pharmacist from a tertiary hospital with years of experience and an intermediate title visited county hospitals, township hospitals, and village clinics where participants were located to inspect pharmacies and medicines with a professional perspective during the interview, as shown Fig.  1 .

Flow chart of study design

Data collection and tools

The quantitative study consisted of three parts. In Part A, a structured questionnaire was distributed to all county hospitals and township hospitals within the CMCs with the support and assistance of Shandong Provincial Health Commission. Structured questionnaires included: basic information on medical institutions, medicines supply, pharmacy services, etc. (Appendix 1 ); in Part B, the study employed a multi-stage stratified random sampling method to randomly select 6 cities in Shandong province based on the level of economic development (high, medium, and low) and geographical location (west, east, and central). For each city, we randomly chose three counties, and then chose three township hospitals within each county. Finally, a total of 35 township hospitals agreed to cooperate with our survey. Two physicians or general practitioners were selected from each township hospital. After the researcher conducted uniform training and pilot testing carried out in a township hospital, a patient survey was carried out on the physician's outpatient shift without any bias in patient selection. Sequentially, medication information from 100 outpatients was recorded, beginning with the first patient. The questionnaire includes: basic characteristics, medication information and medicine shortages for patients (Appendix 2 ); in Part C, the same multi-stage sampling method mentioned in Part B was used to conduct prescription reviews. Ultimately, 37 township hospitals agreed to participate. In each of the 37 township hospitals, prescriptions were randomly selected from a day in mid-July 2023, with 100 prescriptions in order of number of prescriptions in sequence, without any selection bias. Three clinical pharmacists with clinical professional pharmacist qualification, intermediate title or above, and many years of work experience from tertiary medical institutions in Shandong Province were commissioned to conduct prescription reviews in strict accordance with the Prescription Management Measures.

In qualitative study, purposive sampling was used to select 4 out of 16 cities in Shandong Province, and for each city, a CMC was carefully selected to conduct semi-structured interviews with a county hospital, two township hospitals, and four village clinics under its administrative jurisdiction. The participants were pharmacy managers, physicians, village doctors, and a randomly selected patient who purchased medicine with informed consent during interviews at all levels of the CMCs. All interviews were conducted and recorded in a quiet room in the healthcare facility. Interviews lasted an average of 30 mins (25 to 40 mins). The inclusion criteria for the participants were shown in Table  1 . The semi-structured interview guide was developed by clinical pharmacy and public health experts to guide discussions, exploring their understanding of the current state of PHC medicines supply and pharmacy services; whether the supply of PHC medicines truly met patients’ medicines needs and the reason for this (Appendix 3 ). Semi-structured interviews were conducted by two researchers, one responsible for interviews and the other for audio recording, transcription and time management. To ensure the high quality of the interviews, both researchers were trained in qualitative research and several simulation exercises were conducted before the formal interviews.

Data analysis

In quantitative study, all questionnaire information was double-entered by two researchers using Epidata 3.1 to ensure data completeness and accuracy. Prescription reviews were conducted by two clinical pharmacists simultaneously, and in the event of disagreement, a third party was consulted to ensure consensus was reached, while the results of the reviews were entered using Excel 2017 and verified by another researcher to ensure the accuracy of the data. Count data were described by frequencies and percentages. Tests indicated that the measurement data satisfied the normal distribution, described by means and standard deviations (SD), otherwise, median and interquartile range (IQR).

In qualitative study, recordings were made and converted to text using iFLYTEK software (version SR502, KDDI Ltd., Shenzhen, China). They were then imported into NVivo (version 10) for analysis after assessment by two researchers to ensure accuracy. Two researchers (ZF, TG) analyzed the data for thematic framing based on conceptual content analysis and Colaizzi's method, respectively, to minimize subjective bias and error to ensure consistency in results coding [ 31 , 32 ]. There were six steps to the data analysis: (1) two researchers seriously examined each interview transcript many times to fully comprehend it; (2) they critically read and scrutinized the transcripts, highlighting sentences and phrases directly associated with the research objectives; (3) the researchers encoded the significant meanings formulated in statements into distinct categories; (4) the categories were grouped into themes, with clusters of themes related to specific questions forming a new theme; (5) each theme was exhaustively described; (6) Post-review, the descriptions were refined for clarity. Any discrepancies in coding were resolved through discussion until consensus was attained. In this study, participant quotations were identified using the coding system. Each quotation was first labelled with the participant number (1–68), followed by the the participant’s role (PM = pharmacy manager, PH = physician, VD = village doctor, PA = patient) and finally, the participant’s workplace (CH = county hospital, TH = township hospital, village clinic = VC). For example, the 1.PM.CH signified a quotation from a pharmacy manager in a county hospital. Data analysis and reporting of qualitative research followed the consolidated criteria for reporting qualitative research (COREQ) checklist [ 33 ].

Quantitative study

In Part A, questionnaires were returned from 211 county hospitals and 1,581 township hospitals, covering more than 95% of CMCs in Shandong Province. County hospitals were equipped with an average of 774 varieties of medicines, of which 60.0% were essential medicines; township hospitals were equipped with an average of 305 varieties of medicines, of which 76.5% were essential medicines, as shown in Table  2 .

The median annual frequency of medicine shortages in county hospitals was higher than in township hospitals; the median frequency of medicine procurement in county hospitals was 4.0 days, the same as in township hospitals. And the median of medicine arrival rate in county hospitals was four percentage points higher than in township hospitals, as shown in Table  3 .

Annual frequency of medicine shortages was based on the medical institution to fill the National Medicine Shortage Reporting System.

Medicine arrival rate = monthly average number of medicines actually arriving/number of medicines on purchase order × 100%;

Medicine arrival time is the monthly average time between the issuance of a purchase order for medicines and the actual delivery of medicines.

According to the results of questionnaire for CMCs, the average percentage of pharmacists with undergraduate degrees or above in county hospitals was 75.3%, compared to 59.3% in township hospitals. Moreover, the educational level of the pharmacists in different medical institutions varied greatly. There were 610 township hospitals (38.6%) with more than 75 per cent of undergraduate degrees and above, while 252 township hospitals (15.8%) had less than 25 per cent. The number of institutions where county hospitals assisted township hospitals in CMC with patient medication counselling and prescription review was 79 (37.5%) and 88 (41.7%), respectively; the number of institutions actually carrying out rotation work at the upper and lower levels was less than 10%.

In Part B, excluding missing information and unqualified questionnaires, a total of 6,323 medication questionnaires were obtained from PHC patients, 51.2% of whom were female and the mean age of patients was 59.6 ± 17.2 years. After excluding 152 patients not involved in medication use, 945 patients (15.3%) reported this medicine shortages, 52.8 per cent of which was due to the facilities not having the required medicines, usually related to the most common diseases such as diabetes, hypertension and coronary heart disease.

In Part C, 125 prescriptions were excluded due to unclear scanning of paper prescriptions or non-medicine prescriptions, we finally conducted prescription reviews on a sample of 3,575 prescriptions. The prescription qualified rate was only about 72.2%, while the average self-reported prescription qualified rate of county hospitals and township hospitals in Part A was 93.8% and 94.2%, respectively.

Qualitative study

The study interviewed a total of 28 physicians, 12 pharmacy managers and 28 patients. Of the 12 pharmacy managers, 58.3% were women, 75.0% had worked for more than 20 years, and 83.3% had received undergraduate degrees or above. Of the 28 physicians, 17.8% had worked for less than 10 years, and 53.6% had received high or technical school-level education. Of the 28 patients, 64.3% of patients were female, and 71.4% had received junior high school or below education. The mean age of the patients was 53.8 ± 15.2 years, as shown in Table  4 .

Four final themes emerged during the data analysis: (1) Supply of PHC medicines; (2) Solutions to the shortage of off-list medicines; (3) Appropriateness of PHC medicines list; (4) Pharmacist workforce development and pharmacy services. These themes are described in more detail as follows.

Theme 1. Supply of PHC medicines

This theme included two sub-themes that emerged in interviews with pharmacy managers from county and township hospitals and village doctors, including 1) relatively stable supply of PHC medicines based on the existing medicines list; 2) shortage of off-list medicines.

The pharmacy managers in county and township hospitals reported the supply of PHC medicines based on the existing medicines list has been relatively stable and medicine shortages rarely occurred based on the medicine list last year, with shortages of short duration. Even when shortages did occur, they rarely resulted in patients not being able to get their medicines in the existing medicines list when the physicians made prescription, due to the timely checking of pharmacy stocks and advance purchasing plans. Most of the village doctors were very close to the township hospitals and it was very convenient for them to purchase medicines, which were the essential medicines for common and chronic diseases. There were no shortage in the villages.

Medicine shortages are uncommon, but have also occurred a few times in the first few months of this year. Perhaps the medicines of raw materials was out of stock, and then it did not take long to recover, with negligible impact because of the reserve of medicines. (3.PM.TH). There is no shortage of medicines in our village clinic, and as far as I know, there is no shortage in the surrounding villages either. Most of the medicines we purchase are for chronic diseases, common diseases, and there is no shortage of them. My house is very close to it, and I usually drive as soon as I see we are running out of medicines. (38.VD.VC).

The shortage of off-list medicines based on health outcomes has challenged the supply of PHC medicines. Some pharmacy managers reflected that some patients were referred to PHC by higher-level hospitals for medicine, but PHC did not have such medicines in their medicine list, resulting in a kind of "shortage" of off-list medicines.

In reality, some patients presented us with medicines prescribed by higher-level hospitals that we lack. (6.PM.TH)

Theme 2: Solutions to the shortage of off-list medicines

Three themes were included within this theme: (1) recommending alternative medicines; (2) purchasing the required medicines for the patient separately; (3) go to the social pharmacy.

Qualitative results showed when physicians encountered patients who need medicines that were not equipped in the institution, they usually recommended alternative medicines with similar efficacy and price to the patients based on their professional knowledge and clinical experience.

In such instances, we encourage patients to substitute medicines that exhibit comparable clinical efficacy and affordability. Should patients persist, procurement falls to them. (23.PH.CH).

Some healthcare institutions would often purchase the required medicines for the patient separately as a matter of professional ethics.

"For those requiring off-list medicines, we will purchase them individually. They need to wait a few days if they want. (15.PM.TH)

Physicians lacked confidence in their own diagnostic and therapeutic skills, believed that patients were more likely to follow prescriptions from physicians in higher-level hospitals, and were reluctant to recommend alternative medicines to patients, asking them to go to social pharmacies to buy.

Theme 3: Appropriateness of PHC medicines list

This theme consisted two sub-themes through qualitative findings analysis: (1) single variety of PHC medicines; (2) medicines list not updated in a timely and dynamic manner. More details were described as follows:

The appropriateness of the medicines list needed to be optimized. Some of the medicines currently available in health facilities were not the best. And the unmet medication needs of patients occurred in PHC. There was a single variety of medicines in their institutions, so that the use of medicines in clinical practice was limited, and sometimes the use of medicines was not based on the actual health needs of the patients. Most of the available PHC medicines were essential medicines to ensure affordability for patients, and some high-quality but slightly more expensive medicines were unlikely to be available in PHC. However, sometimes patients with chronic disease, especially the elderly suffering from multiple chronic diseases, were in need of these medicines. Many of these elderly people were prescribed medicines in tertiary hospitals in their first visit and they were reluctant to change their medicines easily for fear of poor efficacy.

The supply of PHC medicines can basically meet patients' needs for medicines, but sometimes a better medicine should be available for patients, but hospitals are not equipped with it. (31.PH.TH). Our clinical use of medicines still feels limited. There are some good medicines that are not available in PHC. There are some elderly people with multiple chronic diseases who actually need them. They are very concerned about the efficacy of the medicine if a replacement is recommended (30.PH.TH).

The medicines list was not updated in a timely and dynamically. The physicians after the rotation within the CMCs had revealed certain differences in the medication habits of physicians between the upper and lower levels of hospitals, and there was a certain lag in the equipment and adjustment of medicines in PHC.

Some doctors revealed after the rotation that some medicines had not been used in superior hospitals for a long time, while we still use them, and asked us to change them. (7.PM.CH).

Upon field investigation at healthcare institutions, senior pharmacists reported poor appropriateness of medicines within the medicine supply lists for PHC. Certain medicines, such as compound reserpine tablets and ribavirin injection solution, were removed from professional treatment guidelines or expert consensus. However, these obsolete medicines continued to be favored by PHC.

Theme 4. Pharmacist workforce development and pharmacy services

Two sub-themes were included in this theme during the data analysis (as described above): (1) inadequate capacity for pharmacists in PHC; (2) hoping for setting a communication mechanism.

PHC capacity remained inadequate. Pharmacists claimed a necessity for enhanced skills, citing inadequate grasp of recent advancements in clinical medicine, medicine policy changes and inconsistent timely alterations in pharmaceutical application.

Our ability to obtain information is poor, we know little about those clinical guideline updates and national policy changes, and we don't know the medication habits of doctors in higher hospitals. (8.PM.TH).

Pharmacists in PHC hoped for collaboration with superior hospital for guidance and support.

We sincerely hope that there is a mechanism for communication with higher hospitals, either they come down to give us guidance or we go up to ask them for advice and learning, otherwise we can only judge based on our own experience and ability. (10.PM.TH).

The study found that the supply of PHC medicines based on the existing medicines list was relatively stable, and shortages occurred sporadically and locally. However, off-list medicines were in short supply, failing to meet patients' needs as determined by health outcomes, especially for chronic diseases, which are becoming increasingly prominent. In addition, existing medicine supply lists for PHC are poorly adapted, and medicines are not updated and optimized. The development of pharmacist teams and primary service capacity is inadequate and needs to be further strengthened.

In China, the definition of medicine shortage is a clinically necessary and irreplaceable or incompletely replaceable medicine that has been approved for marketing, and is in insufficient or unstable supply for a certain period of time or region [ 34 ]. There is a well-established premise that all of these come from the medicine list of medical institutions. They are uniformly submitted to the National Medicine Shortage Reporting System (NMSRS), which is often the most important concern of government policy-makers. Implementing strategies that incorporate dynamic, consistent, and prompt surveillance and early warning, augmented by coordinated reactions, has achieved notable progress, including the development of an integrated platform for acquiring medicine shortage data, a shortage-list management system, and a hierarchical and classified disposal system [ 35 , 36 , 37 , 38 , 39 ].

However, there is another potential shortage: off-list medicines were in short supply, failing to meet patients' needs as determined by health outcomes, which is beginning to challenge our existing supply of PHC medicines. In China, this potential shortage is reflected in PHC in the form of a mismatch between upper and lower levels of medication, where patients are diagnosed at a higher level of care and return to a lower level for their medicines, only to find the required medications unavailable. Several studies have shown that this phenomenon occurs in other parts of China and is a growing concern for government policy-makers [ 40 , 41 , 42 ]. It is worth noting that even if there is an adequate supply of medicines on the medicines list of healthcare institutions, which means there is no shortage of medicines, when healthcare institutions do not equip the medicines needed by patients, there is still the embarrassing dilemma of “shortage from the patient's perspective”. Such potential shortages are often invisible and difficult to detect. This is because when a patient has an unmet need for medication, they often choose to replace the medicine or return to the higher hospital to receive it in the clinic. For the former, this strategy is often appropriate for emergency patients, but for patients who require long-term medication, the uncertainty of the efficacy of substitute medicines and medication qualification requires us to seriously consider this behavior; for the latter, PHCs are unable to retain patients because of unavailable medicines for patients, and the resulting outflow of patients runs counter to the national reform objective of promoting hierarchical diagnosis and treatment, which is not what we want to see.

Additionally, another overlooked yet profound issue calls for reconsideration—is the current list of PHC medicines optimally fitted to present needs? Our study found that the appropriateness of medicines on the PHC medicines list is poor. Feedback on the need for common medicines suggests that some have been removed from the list by treatment guidelines or expert consensus in the relevant specialty, but some primary care providers are still accustomed to using them. We hypothesized that potential health effects exist for patients, as no studies have been conducted and adequate empirical evidence is lacking. These issues force us to rethink the functional role in the supply of PHC medicines, which can be transformed from "affordable and accessible" to "better supplied and used" for common and frequent diseases and chronic diseases. This is also in line with the development of hierarchical diagnosis and treatment in China, which promotes the sinking of patients into PHC.

There are two key issues here: first, the development and adaptation of the national essential medicines list (EML); and second, the development and adaptation of the medicines supply list for medical institutions. For the former, the national EML is the cornerstone and an important reference for medical institutions at all levels to select their own medicines supply list, which is directly related to the accessibility and affordability of medicines for patients; for the latter, medical institutions have become the 'last mile' in terms of being able to deliver appropriate medicines to patients. In the past, we have tended to focus on the safety, efficacy, accessibility and affordability of medicines in the existing medicines list, while ignoring the gap between the medicines list and the real needs of patients. PHC medicine supply lists are typically based on the national EML, and we need to rethink whether EML can truly meet the medicine needs of patients. This dilemma is not unique to China, as studies in Brazil, Sri Lanka, Vietnam have shown that EML did not meet patients' needs, especially for chronic diseases [ 43 , 44 , 45 , 46 ].

Currently, the epidemiology of developing countries is in transition from infectious and parasitic diseases to chronic diseases. However, health systems have not yet adapted sufficiently to cope with the long-term nature of chronic diseases. A study of the availability of medicines in 30 developing countries found that medicines for chronic diseases were less available than for acute diseases, especially in the public sector [ 47 ]. With an aging population and an increasing prevalence of chronic disease, the elderly chronic disease population often has multiple commodities or complications, resulting in a more diverse range of medication needs. Inadequate availability of medicines in PHC may hinder the treatment and control of disease [ 48 ]. This is an enlightenment for the World Health Organization and even countries around the world, especially developing countries, to develop their EML. It is suggested that governments should optimize the supply of medicines through their public health systems, especially prioritizing for chronic conditions, to ensure that people have access to the treatment they need. The medicines used to treat common chronic conditions should be available in sufficient quantities in any health system.

Nowadays, the development and adaptation of medicines supply list for medical institutions is at the stage of self-selection and self-management. It may be a challenge for PHC to adjust their medicines supply list to meet the needs of patients in their service area and to adapt to the local disease spectrum. A survey of the availability of essential medicines at PHCs in Indonesia indicated that the most common reason for the unavailability of medicines was that they were subjectively deemed unnecessary by PHC staffs [ 49 ]. In the process of establishing the CMCs, it is necessary to truly develop a medicines list that meet the medication needs of patients within the CMCs, and to achieve the unification of the medicines list among counties, townships and villages is what we need to work on. China has begun to recognize the problem of the linkage between the upper and lower levels of medication and has proposed to explore the establishment of a unified medicines procurement list within the CMCs [ 50 , 51 ]. It is important to note that the unified medicines list within the CMCs is not simply adding and deleting medicines list of institutions at all levels, but rather an innovation to promote higher-level institutions to drive lower-level institutions to optimize and improve the level and habits of medication, so as to achieve homogeneous articulation of medication within the county.

It is crucial to recognize that front-line health care providers, including pharmacists, are the ultimate executors of our policies. A persistent weakness in PHC is poor capacity for health service providers. Although this issue has been improved dramatically, it still requires further development [ 25 ]. This highlights the importance of pharmacy services. A study from Brazil suggests that the low level of implementation of pharmacy services may be closely related to the low availability of PHC medicines [ 52 ]. Under the construction of CMCs, a mechanism of assistance and exchange between medical institutions at the upper and lower levels should be really established. Through substantive exchanges such as pharmacy checkups, medication guidance, and rotation of upper and lower levels, the upper level can lead to an improvement in the capacity of the lower level, and UHC will be achieved early.

This study has several strengths. First, we collected comprehensive information on the supply of PHC medicine from medical institutions in Shandong Province, which are usually not easily accessible, to accurately grasp the current supply of PHC medicine in Shandong Province. Second, we conducted this survey from the perspective of pharmacy managers, physicians and patients at primary healthcare institutions at all levels to multi-dimensional evaluate the supply of PHC medicines and the status of meeting medicine needs. Third, a professional clinical pharmacist visited county hospitals, township hospitals and village clinics where participants were located to inspect pharmacies and medicines from a professional point of view, which is better than most other studies.

Our study also has several limitations. The generalizability of our study is limited by only focusing on Shandong province in China. Our comprehensive literature review has revealed that equivalent scenarios were observed in various Chinese regions. We believe that the outcomes of this research generally illustrate common issues within supply of PHC medicine in China. Second, this study focuses on drawing the attention of government policy-makers and researchers to the gap between essential medicines list and patients' actual medicines needs, but does not provide an in-depth analysis of this gap, which needs to be studied and researched in the future. We believe that this is an important reference for governments, especially in low- and middle-income countries, to formulate essential medicines list.

The supply of PHC medicines based on the existing medicines list has been relatively stable. However, the shortage of off-list medicines based on health outcomes has challenged the supply of PHC medicines. The fundamental reason for this is the mismatch between the actual patient needs for PHC medicine and the existing medicines supply. We need to rethink the focus on the supply of PHC medicines from existing list-driven to prioritizing patient health outcomes to adequately meet patient medicines needs, especially for chronic diseases. Integrated health care provides us with a new way of thinking, by unifying medicine list in the CMCs, breaking down the barriers between medicine list of different levels of medical institutions to meet the needs of patients in the region, and strengthening the capacity of PHC, so as to effectively promote people's health and achieve the UHC early.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Sustainable development goals

Universal health coverage

Primary health care

National volume-based procurement

Integrated health services

County Medical Communities

Non-communicable diseases

Consolidated criteria for reporting qualitative research

Interquartile range

Standard deviations

National Medicine Shortage Reporting System

Essential medicines list

United Nations. SDG's Sustainable Development Goals. 2015. https://www.un.org/zh/59%20104/page/82393 . Accessed Feb 25, 2024.

World Health Organization. Primary health care. 2023. https://www.who.int/health-topics/primary-health-care#tab=tab_1 . Accessed Feb 25, 2024.

Ministry of Health Minister of Health Decision Number HK. Indonesian National Essential Medicine List. 2019.  https://farmalkes.kemkes.go.id/unduh/kepmenkeshk%20-01-07-menkes-688-20192019daftar-obat-esensial-nasional . Accessed June 25, 2024.

Organización Panamericana de la Salud. Servicios farmacêuticos basados en la atención primaria de salud: documento de posición de la OPS/OMS. 2013. http://www.paho.org/hq/index.php?option=com_docman&task=doc_view&gid=21582&Itemid=270 . Accessed June 25, 2024.

Department of Health KZN (2016a). Province of KwaZulu-Natal. Launch of the Central Chronic Medication Dispensing and Distribution Programme. 2016.  http://www.kznhealth.gov.za/mediarelease/2016/Launch-CCMDD-10062016.htm Accessed June 25, 2024.

Ministry of Health National Development and Reform Commission Ministry of Industry and Information. Implementation opinions on establishing the National essential medicine policy. 2009. http://www.gov.cn/ztzl/ygzt/content_1661112.htm . Accessed Feb 25, 2024.

General Office of the State Council. The General Office of the State Council on further improving the shortage of medicines Opinions on the work of ensuring supply and stabilizing prices. 2019. https://www.gov.cn/zhengce/zhengceku/2019-10/11/content_5438499.htm Accessed Feb 25, 2024.

National Health and Family Planning Commission. Ministry of Finance State Commission Office of Public Sectors Reform. Notice on comprehensively promoting the comprehensive reform of public hospitals. 2017. http://www.nhc.gov.cn/cms-search/xxgk/getManuscriptXxgk.htm?id=0563e06eff4441ffa9772dc30b487848 . Accessed Feb 25, 2024.

Office of State Council. Notice on organizing and using the medicines from national bulk procurement. 2019. https://www.gov.cn/zhengce/content/2019-01/17/content%205358604.htm . Accessed Feb 25, 2024.

National Healthcare Security Administration. Opinions on the matching strategies of insurance for the National bulk procurement for medicine. 2019. http://www.nhsa.gov.cn/art/2019/3/5/art_37_952.html . Accessed Feb 25, 2024.

Healthcare Reform Office of the State Council. National Health and Family Planning Commission, National Food and Drug Administration. Opinions on the implementing the dual Invoice for the bulk procurement of medicine in public hospitals. 2016. https://ggzy.guizhou.gov.cn/ztzl/yycgzc/202003/t20200327_556718/22.html . Accessed Feb 25, 2024.

Yuan J, Li M, Jiang X, Lu ZK. National Volume-Based Procurement (NVBP) exclusively for insulin: towards affordable access in China and beyond. BMJ Glob Health. 2024;9(1):e014489.

Article   PubMed   PubMed Central   Google Scholar  

Yang L, Liu C, Ferrier JA, Zhang X. Organizational barriers associated with the implementation of national essential medicines policy: a cross-sectional study of township hospitals in China. Soc Sci Med. 2015;145:201–8.

Article   PubMed   Google Scholar  

General Office of the State Council. Guiding Opinions of the General Office of the State Council on Promoting the Construction of a Hierarchical Diagnosis and Treatment System. 2015. https://www.gov.cn/zhengce/content/2015-09/11/content_10/158.htm . Accessed Feb 25, 2024.

General Office of the State Council, The Guiding Opinions on Promoting the Construction and Development of Medical Communities. 2017. https://www.gov.cn/zhengce/content/2017-04/26/content_5189071.htm . Accessed Feb 25, 2024.

National Health Commission. Notice on Promoting the Construction of a Close-knit County Medical Community. 2019. https://www.gov.cn/zhengce/zhengce/ku/2019 Accessed Feb 25, 2024.

National Health Commission. Guiding Opinions on Comprehensively Promoting the Construction of Close-knit County Medical and Healthcare Communities. 2023. https://www.gov.cn/zhengce/zhengceku/202312/content_6923447.htm Accessed Feb 25, 2024.

Wang N, Yang Y, Xu L, Mao Z, Cui D. Influence of Chinese national centralized drug procurement on the price of policy-related drugs: an interrupted time series analysis. BMC Public Health. 2021;21:1–10.

Article   Google Scholar  

Rong X, Yin J, Duan S, Sun Q, Babar ZU. The effects of pricing policy on the prices and supply of low-cost medicines in Shandong, China: evidence from an interrupted time series analysis. BMC Public Health. 2020;20(1):588.

Yang Y, Tong R, Yin S, Mao L, Xu L, Hao S, et al. The impact of “4 + 7” volume-based drug procurement on the volume, expenditures, and daily costs of antihypertensive drugs in Shenzhen, China: an interrupted time series analysis. BMC Health Serv Res. 2021;21:1275.

World Health Organization. Progress report on the United Nations decade of healthy ageing, 2021–2023: executive summary. 2023. https://iris.who.int/handle/10665/374302 . Accessed Feb 25, 2024.

World Health Organization. Noncommunicable diseases. 2023. https://www.who.int/health-topics/noncommunicable-diseases#tab=tab_1 . Accessed Feb 25, 2024.

National Bureau of Statistics. Statistical bulletin on national economic and social development in 2023. 2024. https://www.gov.cn/lianbo/bumen/202402/content_693/4935.htm Accessed Apr 25, 2024.

National Health Commission. The 14th Five-Year Plan for Healthy Aging. 2022. https://www.gov.cn/gongbao/content/2022/content_5692863.htm Accessed Apr 25, 2024.

Li X, Lu J, Hu S, Cheng KK, De Maeseneer J, Meng Q, et al. The primary health-care system in China. Lancet. 2017;390(10112):2584–94.

National Food and Drug Administration. Notice on the implementation of opinions from office of state Council on establishing the quality and bioequivalence assessment on generic medicines. 2016. https://www.nmpa.gov.cn/directory/web/nmp%20a/xxgk/ggtg/qtggtg/20160526165701188.html . Accessed Feb 25, 2024.

Office of State Council. Opinions on establishing the quality and bioequivalence assessment on generic medicines. 2016. https://www.gov.cn/zhengce/content/2016-03/05/content5049%20364.htm . Accessed Feb 25, 2024.

National Health Commission. Opinions on Strengthening Pharmaceutical Management in Medical Institutions to Promote the Rational Use of Medicines. 2020. https://www.gov.cn/gongbao/content/2020/content_5522549.htm Accessed Apr 25, 2024.

Shandong Provincial Bureau of Statistics Population. 2023. http://www.shandong.gov.cn/art/2023/7/5/art_98094_597183.html . Accessed Feb 25, 2024.

Hao W, Lin Z, Xiaoya F, Ruyue D, Jun Y. Study on the current status of chronic disease co-morbidity and its spatial distribution among middle-aged and elderly people in China. Chinese Family Med. 2022;25(10):1186–90.

Google Scholar  

Content Analysis Method and Examples Columbia Public Health. https://www.publichealth.columbia.edu/research/population-health-methods/content-analysis (2016). Accessed Feb 25, 2024.

Wirihana L, Welch A, Williamson M, Christensen M, Bakon S, Craft J. Using Colaizzi’s method of data analysis to explore the experiences of nurse academics teaching on satellite campuses. Nurse Res. 2018;25(4):30–4.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19:349–57.

National Health Commission. Notice of the General Office of the National Health Commission on Printing and Distributing the Technical Guidelines for the Classification, Grading and Alternative Use of Drugs in Shortage in Medical Institutions. 2019. https://www.gov.cn/xinwen/2019-07/29/content_5416151.htm . Accessed Feb 25, 2024.

Office of Ministry of Health Office of Ministry of Industry and Information. Opinions on ensuring the supply of essential medicines for infectious diseases and emergency medicines. 2011. http://www.nhc.gov.cn/yaozs/s3581/201112/6554087caea4493f8ff5297c93ef9c10.html . Accessed Feb 25, 2024.

Ministry of Industry and Information Ministry of Health National Development and Reform Commission. Notice on the designated manufacture for essential medicines with high risk of shortage. 2012. http://www.nhc.gov.cn/zwgk/wtwj/201304/46934b151933472baf628f953d540227.html . Accessed Feb 25, 2024.

National Health and Family Planning Commission Ministry of Industry and Information National Development and Reform Commission. Notice on establishing designated manufacturer for clinical necessary and shortage in supply medicines. 2016. http://www.nhc.gov.cn/yaozs/s3581/201701/75150182dbb74ca78ce3397d28/d51cb8.html . Accessed Feb 25, 2024.

National Health and Family Planning Commission National Development and Reform Commission Ministry of Industry and Information. Notice on promoting the supply of shortage medicine. 2017. http://www.nhc.gov.cn/yaozs/s3581/201706/e/c2c5910ba4341559e0176e5e2/37d5e6.html . Accessed Feb 25, 2024.

Mao W, Jiang H, Mossialos E, Chen W. Improving access to medicines: lessons from 10 years of drug reforms in China, 2009–2020. BMJ Glob Health. 2022;7(11):e009916.

Wu D, Lam TP, Lam KF, Sun KS, Zhou XD. Patients’ coping behaviors to unavailability of essential medicines in primary care in developed urban China. Int J Health Policy Manag. 2021;10(1):14–21.

PubMed   Google Scholar  

Zhao M, Jia D, Yong C, Su Q. Facilitating the articulation of medication use in upper and lower level healthcare organizations with the base drug system based on the governance perspective. Health Soft Sci. 2022;36(08):24–9.

Rui Z, Xiuyuan S. Dilemma analysis and countermeasure suggestions on the convergence of upper and lower medication in China’s medical consortium. China Health Econ. 2022;41(11):13–6.

Nascimento RCRMD, Álvares J, Guerr AA Jr, Gomes IC, Costa EA, Leite SN, et al. Availability of essential medicines in primary health care of the Brazilian Unified Health System. Rev Saude Publica. 2017;51(suppl 2):10s.

Karnikowski MGO, Galato D, Meiners MMMA, Silva EVD, Gerlack LF, Bós ÂJG, et al. Characterization of the selection of medicines for the Brazilian primary health care. Rev Saude Publica. 2017;51(suppl 2):9s.

Rathish D, Premarathna I, Jayathilake T, Kandegedara C, Punchihewa K, Ananda L, et al. Availability of essential medicines in selected public, primary and secondary health care institutions of a rural Sri Lankan district: a spot survey. BMC Health Serv Res. 2017;17(1):11.

Hoa NT, Derese A, Peersman W, Markuns JF, Willems S, Tam NM. Primary care quality in Vietnam: Perceptions and opinions of primary care physicians in commune health centers: a mixed-methods study. PLoS ONE. 2020;15(10):e0241311.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Cameron A, Roubos I, Ewen M, Mantel-Teeuwisse AK, Leufkens HG, Laing RO. Differences in the availability of medicines for chronic and acute conditions in the public and private sectors of developing countries. Bull World Health Organ. 2011;89(6):412–21.

Su M, Zhang Q, Bai X, Wu C, Li Y, Mossialos E, et al. Availability, cost, and prescription patterns of antihypertensive medications in primary health care in China: a nationwide cross-sectional survey. Lancet. 2017;390(10112):2559–68.

Fanda RB, Probandari A, Yuniar Y, Hendarwan H, Trisnantoro L, Jongeneel N, et al. The availability of essential medicines in primary health centres in Indonesia: achievements and challenges across the archipelago. Lancet Reg Health Southeast Asia. 2024;22:100345.

National Health Commission. Bureau of Traditional Chinese Medicine. Measures for the Administration of Medical Consortiums (for Trial Implementation). 2020. https:// www.gov.cn/gongbao/content/2020/content_5554582.htm?ivk_sa=1024320u . Accessed Feb 25, 2024.

Notice on Printing and Distributing Key Tasks for Deepening the Reform of the Medical and Health System in the Second Half of 2023. 2023. https://www.gov.cn/zhengce/zhengceku/202307/content_6894073.htm . Accessed Feb 25, 2024.

Pereira NC, Luiza VL, Campos MR, Chaves LA. Implementation of pharmaceutical services in Brazilian primary health care: a cross-sectional study. BMC Fam Pract. 2021;22(1):170.

Download references

Acknowledgements

The authors thank the staff in Shandong Provincial Health Commission for helping to collect the data used in this study.

This work was supported by the National Key Research and Development Program of China (NO. 2020YFC2006505).

Author information

Zhixin Fan and Tiantian Gao have contributed equally to the work.

Authors and Affiliations

Centre for Health Management and Policy Research, School of Public Health, Cheeloo College of Medicine, Shandong University, Jinan, 250012, Shandong, China

Zhixin Fan, Tiantian Gao & Qiang Sun

NHC Key Lab of Health Economics and Policy Research (Shandong University), Jinan, 250012, Shandong, China

Shandong Provincial Hospital affiliated to Shandong First Medical University, Jingwu Road, Jinan, 250021, Shandong, China

Tiantian Gao

Centre for Pharmaceutical Policy and Practice Research Department of Pharmacy, University of Huddersfield, Queensgate, Huddersfield, HD1 3DH, UK

Zaheer-Ud-Din Babar

You can also search for this author in PubMed   Google Scholar

Contributions

ZF, TG, ZB and QS developed the study concept. ZF, TG and QS designed the research and collected data. ZF and TG analyzed the data and drafted the manuscript. QS and ZB revised the manuscript critically for important intellectual content. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Qiang Sun .

Ethics declarations

Ethics approval and consent to participate.

Ethics was reviewed and approved by the Ethics Committee of the Centre for Health Management and Policy Research, Shandong University (NO.ECSHCMSDU20231201).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests. Qiang Sun is an Editorial Board member of Global Health Research and Policy and Qiang Sun was not involved in the review process for the decision of the manuscript.

Supplementary Information

Additional file 1: appendix 1. structured questionnaire for medical institutions in the cmcs., additional file 2: appendix 2. medication questionnaire for patients in phc., additional file 3: appendix 3. outline of interview with pharmacy manager, physicians and patients., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Fan, Z., Gao, T., Sun, Q. et al. Whether medicine supply is really meeting primary health care needs: a mixed-methods study in Shandong Province, China. glob health res policy 9 , 32 (2024). https://doi.org/10.1186/s41256-024-00374-x

Download citation

Received : 18 March 2024

Accepted : 02 August 2024

Published : 05 September 2024

DOI : https://doi.org/10.1186/s41256-024-00374-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medicine supply
• Medicine need
• Medicine shortage
• Mixed-methods study

Global Health Research and Policy

ISSN: 2397-0642

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 02 September 2024

Empathy ability and influencing factors among pediatric residents in China: a mixed-methods study

• Pingping Li 1   na1 ,
• Ling Weng 2   na1 &
• Lu Dong 1  

BMC Medical Education volume  24 , Article number:  955 ( 2024 ) Cite this article

Metrics details

Empathy is one of the fundamental factors enhancing the therapeutic effects of physician–patient relationships, but there has been no relevant research in China on the pediatric resident physicians’ capacity for empathy or the influencing factors.

A mixed-methods study was undertaken. The student version of the Jefferson Scale of Empathy was used to assess 181 postgraduate residents at Shanghai Children’s Medical Center and Shanghai Children’s Hospital. Differences in empathy ability among pediatric resident physicians of different genders and specialties were analyzed using independent sample t-tests and Mann–Whitney U tests. A one-way analysis of variance was used to analyze the differences in empathy ability at different educational levels and years of medical residency training. Seven third-year postgraduate pediatric residents from Shanghai Children’s Medical Center participated in semi-structured interviews exploring the influencing factors. We analyzed the interview transcripts using thematic analysis.

The scale was completed by 154 pediatric residents. No statistically significant differences in empathy were found between educational level, postgraduate year, gender, or specialty. The factors influencing empathy in doctor–patient communication included the person who accompanied the child to see the doctor, how the children cooperated with doctors for medical treatment, the volume of pediatric outpatient and emergency visits, and the physician’s ability to withstand pressure. All interviewed resident physicians regarded learning empathy as important but rarely spent extra time learning it.

Conclusions

The evaluation results of resident physicians on changes in empathy after improving clinical abilities vary according to their understanding of empathy, and the work environment has an important impact on pediatricians’ empathy ability. Their empathy score is relatively low, and this requires exploration and intervention.

Peer Review reports

There has been a long-standing tension in the physician–patient relationship in pediatric clinics in China [ 1 ]. There are complex reasons for this, but research has found that 80% of doctor–patient disputes result from poor communication, often due to a lack of empathy during interactions [ 2 , 3 ]. The current medical literature defines empathy as the ability to understand the patient’s perspective and feelings, as well as sharing and acting on this understanding during interpersonal interactions [ 4 ]. Studies show that empathy is linked with enhanced patient satisfaction and treatment compliance [ 5 ]. High levels of empathy in healthcare professionals are connected to positive clinical prognoses for patients by reducing mental stress, improving self-awareness, and reducing anxiety and depression [ 6 , 7 ].

Residency training is mandatory for doctors to qualify to practice independently [ 8 ]. In China, standardized residency training began nationwide in 2013; seven government ministries jointly issued the policy document, “Guidance on the Establishment of a Standardized Residency Training System” [ 9 ]. All clinicians, including pediatricians, are required to undergo three-year residency training after graduating from medical school. During these three years, residents study in different departments.

The Chinese Medical Doctor Association recommends six core competencies for medical residents based on the content and standards for standardized residency training (2022 version): professionalism, clinical professionalism, managing patients, communication, teaching, and learning. While professionalism necessarily involves knowledge and skill, the unique characteristic of medical professionalism is empathy [ 10 ], a capacity that is also strongly related to communication. Thus, cultivating empathy is important for medical residents.

The student version of the Jefferson Scale of Empathy (JSE-S) was specifically developed as a self-report scale for the assessment of empathy in medical students [ 11 , 12 ]. Some studies have reported a decline in empathy among medical students [ 13 , 14 , 15 ], while some have noted that students in their final year scored higher for empathy than did first-year medical students [ 16 , 17 ] and others have reported little change in empathy scores across the years [ 18 ]. However, there is little comparable research for China.

Some studies have shown that the work environment can affect the development of empathy [ 19 ], and pediatric departments recorded a high incidence of doctor–patient disputes [ 20 ]. According to the 2019 National Medical Injury Liability Dispute Case Big Data Report, pediatrics is a high-risk area for doctor–patient disputes.

Therefore, this study aimed to analyze whether there are differences in the ability to empathize among pediatric resident physicians of different grades and whether the pediatric medical environment affects that ability. A mixed-methods approach was used: We assessed empathy scores using the JSE-S and then conducted a semi-structured survey to discuss the influencing factors.

Study design

Quantitative and qualitative methodologies were used to analyze empathy and influencing factors among pediatric residents, incorporating a survey for the quantitative analysis and interviews for the qualitative assessment.

Quantitative methodology

Data collection: survey.

In July 2023, all residents of the Shanghai Children’s Medical Center, affiliated with Shanghai Jiao Tong University School of Medicine, and the Children’s Hospital affiliated with Shanghai Jiao Tong University School of Medicine, were surveyed using an anonymous online questionnaire. Informed consent was obtained from all participants. The survey was available online for one week, and after three days, the residents were sent reminders via WeChat by staff members from the two hospitals.

The JSE-S was used in this study [ 21 ] The scale consists of 20 items, measured using a seven-point Likert scale ranging from 1 = completely disagree to 7 = completely agree but with items 1, 3, 6, 7, 8, 11, 12, 14, 18, and 19 reverse scored. The total score of the scale comprises the total score for all items, with higher scores indicating higher levels of empathy. The scale is subdivided into three dimensions: perspective-taking, compassionate care, and standing in the patient’s shoes [ 12 , 21 ]. The maximum score on the JSE is 140, and the minimum score is 20. Other data collected as part of the JSE survey included sex and years of medical resident training, specialty, and education.

Data analysis

Independent samples t-tests were performed to assess differences in mean JSE scores between sexes. The Mann–Whitney U test was used to compare the differences in mean JSE scores between specialties. A one-way analysis of variance (ANOVA) was performed to compare the differences between the different years of medical residency training and different levels of education. All analyses were performed using the IBM SPSS Statistics Version 25.0. The data are presented as mean ± standard deviation (SD) unless otherwise stated.

Qualitative methods

Data collection: interviews.

As the third-year postgraduate (PGY3) pediatric residents who entered standardized training for pediatric resident physicians in 2020 had completed their training, in August 2023, PGY3 pediatric residents at the Shanghai Children’s Medical Center were asked to participate in the interviews. Seven consented to participate (Table  1 ).

Two researchers (LPP and WL) conducted individual face-to-face semi-structured interviews. The interviews lasted 50–70 min (60-minute average) and were audio recorded and transcribed verbatim by a professional service. The interview guide (Table  2 ) included three aspects: work environment, residents’ standardized training, and open questions. The open-ended questions explored the most memorable cases of smooth and unsmooth communication with patients.

During the interviews, the research followed the guidelines of the interview outline and interviewees’ actual situations. The order and method of questioning were adjusted according to the context and the value of the questions. The language used by the interviewees was accepted without judgment, and no inducements or interventions were made. To protect the privacy of the respondents, their names have been replaced by numbers.

In accordance with a constructivist approach, the analyses tapped into the sense that the participants made of their experiences of communicating with patients. Inductive thematic analysis [ 22 ] was used to identify themes. The interviews were audio recorded and transcribed verbatim by a professional service (iFLYTEK). WL and LPP read and reread transcripts for immersion and familiarization. Two authors (WL and LPP) iteratively coded the data deemed relevant to the current study using Nvivo14 [ 23 ]. Disagreements were discussed with another author (DL). The next step was to group related codes into potential themes. Subsequently, three authors (LPP, WL, and DL) jointly reviewed the themes to ensure that the codes in each theme were coherent and that the codes in different themes could be clearly distinguished.

Quantitative research results

Study population characteristics.

In total, 154 residents responded to the survey, a response rate of 85.1% (154/181). The participating pediatric residents included 60 (39.0%) residents from postgraduate year 1 (PGY1), 48 (31.1%) from postgraduate year 2 (PGY2), and 46 (29.9%) from PGY3. A total of 111 participants (72.1%) were women, and 43 (27.9%) were men. A total of 112 (72.7%) participants were pediatric residents, and 42 (27.3%) were pediatric surgery residents. There were 63 (40.9%) undergraduate residents, 69 (44.8%) master’s residents, and 22 (14.3%) doctoral degree residents in this study. The mean JSE-S score for the overall study population was 81.41 ± 5.43.

Based on the independent samples t-test and Mann–Whitney test, we found no differences in pediatrics’ sex (t = 0.878, p  = 0.381) or specialty (z=-0.981, p  = 0.327).

The education levels of different residents were not significantly different (f = 1.455, p  = 0.237) (Table  3 ).

Empathy competencies of pediatric residents with different pediatric standardized training years

The empathetic recognition mean JSE-S score was 81.41 ± 5.43. Compared to PGY1 (81.33 ± 4.45) and PGY2 (80.75 ± 4.08), PGY3 had a high JSE-S score (82.2 ± 7.48), but there were no significant differences between different years of medical residency training (f = 0.839, p  = 0.434) (Table  4 ).

In the perspective-taking scale, the mean JSE-S score was 54.66 ± 6.70, and the one-way ANOVA revealed significant differences between PGYs (f = 3.51, p  = 0.032). There were significant differences between PGYs for three items: “Physicians’ understanding of the emotional status of their patients, and that of their families is an important component of the physician–patient relationship” (f = 4.391, p  = 0.014); “Physicians should try to stand in their patients’ shoes when providing care to them” (f = 4.697, p  = 0.010); and “I believe that empathy is an important therapeutic factor in medical treatment” (f = 250.996, p  = 0.000).

The mean JSE-S score on the compassionate care scale was 20.76 ± 5.97. PYG1, PYG2, and PYG3 scored 22.42 ± 4.48, 19.42 ± 6.17, and 20.00 ± 7.00, respectively, indicating significant differences between them (f = 4.053, p  = 0.019). Significant differences were found for years of pediatric residency training for “Physicians should not allow themselves to be influenced by strong personal bonds between their patients (f = 40.158, p = 0.000) and their family members” and “I do not enjoy reading non-medical literature or the arts.” (f = 37.236, p  = 0.000).

The standing in the patient’s shoes dimension of the JSE-S showed no significant differences between the PGYs.

Qualitative research results

The influence of pediatric visiting environment on physicians’ empathy ability.

Because children are unable to express their discomfort or illness well, they should be accompanied by parents or grandparents when attending hospital. Doctors, therefore, have to communicate with the parents or grandparents, and their circumstances, including their education level, familiarity with the child, physical health status, communication and understanding skills, and attitude toward doctors, can affect empathy between doctors and patients.

Compared to adult hospitals , the empathy ability of doctors in children’s hospitals may be slightly reduced because we are dealing with parents , not patients themselves , and many of them are brought for treatment by elderly people. Elderly people do not understand the child’s disease or may have difficulty hearing clearly , which can greatly affect communication , let alone empathy. (P1, M) Some elderly people may regard their children’s condition unnecessarily seriously , resulting in us not being able to understand the symptoms of the child properly. (P2, F) Parents tend to have a good understanding of the child’s condition. If grandparents with a low education or if other relatives bring them over , the process of consultation may not be very smooth. (P3, F) The child might be brought over on the first day of treatment by their parents but subsequently by older relatives. Because the child is still running a fever for two or three days , they will be very anxious. When they communicate this to us , their attitude is often poor. (P4, M) If an elderly person brings a child to see a doctor , I often ask the elderly person to call the parents on the spot so I can listen to them. It is better this way. (P7, M)

Some resident physicians said that the language of the patients’ parents significantly impacted their ability to empathize:

Because I am not from Shanghai and grandparents who accompany their children may speak the local dialect , we are unable to communicate. This is challenging for me and many colleagues because most of us cannot understand the Shanghai dialect. (P2, F)

The child’s upbringing and willingness to cooperate with treatment were also identified as important:

Some parents may spoil their children , some children start acting spoiled as soon as they arrive at the clinic , and some even make a scene , which can interfere with the medical treatment. (P2, F)

The volume of pediatric outpatient and emergency visits and the self-regulation ability of physicians facing strong workloads can also affect communication and empathy between doctors and patients:

Outpatient hours may limit our communication with patients. Generally , you need to finish one within 5–10 min. Otherwise , the patient’s visit may be too long , and you may not be able to see all registered patients before leaving work. For example , last summer , our two doctors saw an average of around 130–150 patients a day , while I saw an average of 80–90 patients per day. That was during the pandemic last year , and there will definitely be more this year. (P7, M) The doctor is very tired and has a large number of patients. If the patients are in a hurry , you need to see them within a short period. If our resident physician’s self-regulation ability is not good , it will affect communication. (P5, M)

Standardized training for resident physicians to cultivate empathy skills

The three resident physicians interviewed believed that in their first year of participating in standardized resident training, they felt more empathy for patients due to their lack of clinical knowledge. By contrast, after three years of clinical practice and improvements in their clinical knowledge, they viewed the patient’s condition more rationally and from a medical perspective.

Because you have learned systematic knowledge about diseases , you know what the likely outcome will be objectively. Consequently , your empathy regarding the intermediate treatment process and patients may decrease , and you have to think about the treatment from a doctor’s professional perspective. (P2, F) When I first entered standardized training for resident physicians , I lacked clinical experience and was not familiar with the treatment process for many diseases. When I encountered critically ill patients , I felt that they were so pitiful. After three years of training , however , these diseases have become more familiar. I know the treatment processes for each disease and feel that empathy has decreased. (P3, F)

The two residents felt that empathy followed a curved path. Residents who have just entered clinical practice have relatively high empathy. However, as their clinical abilities and understanding of diseases increase, coupled with the busy workload of clinical work, their empathy decreases. However, empathy may improve after becoming a physician.

When I went to the outpatient clinic with my supervisor , I felt that my supervisor , who was already a chief physician , had reached a very high level of empathy. I think his empathy ability was much stronger than mine; that is , regardless of the patient’s attitude , he could think from the patient’s perspective. As a resident physician , I still cannot reach the level of empathy that my supervisor possesses. Perhaps I need to acquire some experience in my career to reach the level of empathy that my supervisor possesses , but the process may be a bit complex. (P2, F) As a physician , I think that empathy is a curved process , initially high , but as your clinical abilities improve and work experience increases , empathy may decrease. The attending physician is very busy , and at some point , the value of empathy may be underestimated , but it increases again with age. Perhaps at a certain point or stage , you suddenly feel it is important , and you become very focused on the ability to empathize. (P3, F)

Two interviewees believed that after three years of standardized training for resident physicians, their empathy skills had improved. Three years ago, they only thought about the disease. Today, they are able to think from the perspective of the patient and stand in their shoes.

For example , parents who come to the surgical emergency department are very anxious. As a physician , I can understand their feelings. Some common diseases that you have seen before have a likely trajectory. Although you are also anxious about their diseases , you know how to treat different disease symptoms and have the ability to handle them. I know why parents are anxious , and I can think from their perspective. (P4, F) As you gain an understanding of diseases and as your own abilities and clinical experience improve , your feelings toward the patient change. Because I know how a disease like Mycoplasma pneumonia , for example , develops , when I was in PGY1 , I felt that the child’s cough was very severe , which made the parents very anxious. At the time , I was also quite anxious. Now , however , I know that the course of this disease is long. If parents are very anxious , I will explain this disease to them and comfort them. I have had more contact with patients , and I will consider the problem more from their perspective. (P6, F)

Cultivating residents’ empathy ability during standardized resident training

Self-study: The residents believed it important to learn theories relevant to doctor–patient communication and empathy. The interviews revealed that most of them improved their communication skills in clinical practice, and a few residents spent time studying how to communicate with patients. Only one student bought a book about communication, and one student paid attention to the ability to communicate with patients because they had to take an exam on doctor–patient communication.

When I was admitted for training , there was a medical teacher talking about doctor–patient disputes , which was quite scary at the time. I bought relevant books but did not read them. (P1, M) I have not bought any books related to doctor–patient communication , but I think in clinical practice , it is necessary to participate more in the conversation process with superiors , listen more to their conversations , listen more to how they communicate with patients , and then try to learn how to better communicate with patients on my own. (P2, F) This year’s standardized training and graduation assessment for resident physicians added an assessment of doctor–patient communication. I have paid attention to this knowledge , but I have not delved into it. (P3, F)

Training course: It is necessary to set courses to cultivate residents’ empathy ability, such as theoretical training courses, case-sharing groups, and scenario simulations.

I think it’s necessary to set courses for residents to teach us how to communicate , how to express the appropriate level of empathy to patients , etc. (P1, M) I think theoretical teaching in this area is possible , but it cannot be a single output of this teaching mode. Instead , we could hold some doctor–patient communication and sharing meetings , where residents or specialists could share their cases in clinical work and learn from each other . (P3, F) Maybe establish some scenario simulation courses for training. (P5, M)

Sharing the most memorable cases during resident training

Due to the fact that resident physicians undergo rotational training in different clinical departments over 3 years, clinical departments, patient situations, work environments, and severity of diseases may vary. By conducting interviews with resident physicians during the training period, the factors that affect the empathy ability of resident physicians can be further explored by allowing them to profoundly impact the departments where communication with patients is not smooth or smooth. The results are shown in Table  5 .

Clinical empathy and number of years of standardized training

Some studies have shown that empathy scores are associated with ratings of clinical competence [ 24 ]. From the results of the questionnaire survey, the JSE-S scores of PGY1, PGY2, and PGY3 showed no significant differences. From the interview results, seven respondents compared the changes in their empathy skills between the beginning and completion of the standardized resident physician training. Five pediatric resident physicians believed that their empathy skills had decreased with the improvement in their medical skills, while two resident physicians believed that their empathy skills improved after receiving standardized resident physician training. The results of the interviews seem to confirm the results of the questionnaire survey that different physicians have different understandings of the relationship between the improvement of clinical abilities and empathy. These two perspectives may be due to different perspectives on empathy. A resident physician who believes that empathy decreases may believe that the physician’s empathy toward patients is more about the patient’s illness. As their medical abilities improve, they can treat the patient’s illness and believe that it will eventually be cured, so the need for empathy decreases. Some studies have reported that doctors who sympathize with their patients share their suffering, which could lead to emotional fatigue and a lack of objectivity [ 25 ]. However, one resident physician believed empathy had improved by progressing from learning about diseases from books during their medical student stage to the realities of clinical practice, seeing the impact of diseases on patients, families, and even society.

Clinical empathy and the pediatric work environment

Doctor–patient communication in pediatrics is more complex and difficult than when treating adults, meaning that pediatricians bear higher risks. The probability of medical disputes in pediatrics is much higher than in other departments; pediatricians are often insulted and even physically threatened [ 26 ]. Physician empathy is at the heart of doctor–patient communication and significantly influences patient outcomes [ 27 ]. This study explored the factors that influence empathy between pediatricians and patients. In patient terms, the level of cooperation from the child and the characteristics of the person accompanying the child are factors. As for the doctors, they can be confronted with pressure and the need to communicate effectively in the face of high outpatient volumes, which can affect their expressions of empathy, a finding similar to that of previous studies [ 28 , 29 ].

Further analysis of direct doctor–patient communication and empathy among pediatric resident physicians in different rotating departments showed that communication between doctors and patients was seen to be smoother in the Rheumatology and Immunology, General Surgery, and Special Diagnosis Departments, while difficulties were encountered in Outpatients and Emergency, Hematology and Oncology, Surgical Oncology, and Cardiology. The reasons may be complex, but four principal issues can be identified. First, the duration of communication between doctors and patients and the environment of medical treatment; in the Special Diagnosis Department, for example, patients are able to communicate and interact with doctors for a long time, and the medical environment is very good, whereas Outpatients and Emergency see a rapid turnover and high workload. Second, the level of familiarity between patients and physicians can play a role. In Rheumatology and Immunology Departments, for example, there are often patients with chronic diseases who have been hospitalized for a long time; doctors and patients are very familiar with each other, and some studies have shown empathy is easier to generate when closer interpersonal relationships develop [ 30 ]. Third, different teaching methods may have an impact. Better training on the wards can make residents feel more confident in communicating with patients, whereas Outpatients and Emergency can require residents to face patients alone, generating anxiety or even burnout [ 31 ]. Fourth, disease severity can play a role. In some departments, such as Hematology and Oncology, patients may not have a high hope of recovery but may have high expectations of the treatment. This may not only put a lot of pressure on doctors but also make it difficult to communicate effectively with patients; research has indicated that there is still a gap between the actual and expected disclosure of “bad news” about cancer among healthcare workers, patients, and family members, leading to various disclosure dilemmas [ 32 ].

Clinical empathy across different settings

The mean empathy levels found in this study (81.41 ± 5.43) are lower than those reported [ 33 ] in most similar studies around the world. Similar lower JSE scores have been seen in undergraduate medical students in China; the average JSE score among medical students from Sun Yat-sen University was 84 [ 34 ]. This finding is concerning. The shortage of pediatricians, [ 35 ] low wages, [ 36 ] severe occupational burnout, [ 37 ] and the influence of Asian parental culture [ 38 ] may partly explain our findings. Further investigations are required to determine the factors associated with such low scores so that steps can be taken to address the situation.

Cultivating empathy among pediatric residents

Our research shows that resident physicians believe that empathy is important, even though their self-rated empathy scores are less than ideal. Interventions to further investigate the teaching and learning of empathy were discussed [ 39 ]. Many training courses have proven to be beneficial in enhancing the empathy skills of resident physicians. The teaching innovation “How to act-in-role” has been shown to be effective not only in increasing medical students’ self-reported empathy but also in their competence in consultation skills [ 40 ]. The addition of narrative medicine-based education in standardized training improved empathy and may have improved the professional knowledge of residents [ 41 , 42 ] The use of Balint group activities [ 43 ] with residents has shown significant improvements in empathy across all dimensions. Medical schools should design appropriate training courses and implement interventions at all stages (from the admission process to curricula to residency) and levels (explicit and implicit curricula) depending on the empathy levels of their resident physicians.

Our findings suggest that, based on the different understandings of empathy among resident physicians, the clinical empathy level of pediatric resident physicians is not closely related to an improvement in clinical abilities. Rather, the working environment of pediatricians significantly impacts their empathy ability. Empathy is lower among pediatric residents in China when compared to their European counterparts, and further research into the underlying factors associated with such low scores is necessary to plan interventions to cultivate empathy among pediatric residents.

Limitations

One important weakness of this study is that it was based in one medical school with two specialized children’s hospitals; the limited sample size of the investigation and interviews may mean that the study is not representative of pediatric residents in China. Moreover, the cross-sectional survey precluded us from identifying a causal relationship; thus, a prospective longitudinal study with a larger sample size of pediatric residents is warranted.

Data availability

The questionnaire data that support the findings of this study are available in the Baidu Netdisk repository, https://pan.baidu.com/s/1hRjCKuIVVry79HwTzxB_bA with the primary accession code e9hp.The interview datasets analysed during the current study are not publicly available due to privacy concerns but are available from the corresponding author upon reasonable request.

Wenhui G, Xinqing Z, Shanshan L, et al. Cognitive analysis of medical staff in clinical departments of 45 hospitals in nine provinces on the tense doctor-patient relationship. J Southeast Univ. 2018;20(4):124–129145. https://doi.org/10.3969/j.issn.1671-511X.2018.04.014 . Philosophy and Social Sciences Edition.

Article   Google Scholar  

Zhang X, Sleeboom-Faulkner M. Tensions between medical professionals and patients in mainland China. Camb Q Healthc Ethics. 2011;20(3):458–65. https://doi.org/10.1017/S0963180111000144 .

Jiang S. Pathways linking patient-centered communication to health improvement: a longitudinal study in China. J Health Commun. 2019;24(2):156–64. https://doi.org/10.1080/10810730.2019.1587110 .

Mercer SW, Maxwell M, Heaney D, Watt GC. The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Fam Pract. 2004;21(6):699–705. https://doi.org/10.1093/fampra/cmh621 .

Kane GC, Gotto JL, Mangione S, West S, Hojat M. Jefferson scale of patient’s perceptions of physician empathy: preliminary psychometric data. Croat Med J. 2007;48(1):81–6.

Google Scholar  

Lorié áine, Reinero DA, Phillips M, et al. Culture and nonverbal expressions of empathy in clinical settings: a systematic review. Patient Educ Couns. 2017;100:411–24.

Hemmerdinger JM, Stoddart SDR, Lilford RJ. A systematic review of tests of empathy in medicine. BMC Med Educ. 2007;7:24.

Joyce BL, Scher E, Steenbergh T, Voutt-Goos MJ. Development of an institutional resident curriculum in communication skills. J Grad Med Educ. 2011;3(4):524–8.

Li FY, Wen Y, Lei PG, et al. The present situation and consideration of Residency standardized training in China. China Contin Med Educ. 2019;11:92–4.

Montgomery L, Loue S, Stange KC. Linking the heart and the head: humanism and professionalism in medical education and practice. Fam Med. 2017;49(5):378–83.

Alcorta-Garza AJ, Gonzalez-Guerrero JF, Tavitas-Herrera SE, Rodrigues-Lara FJ, Hojat M. Validity of the Jefferson scale of physician empathy among Mexican medical students. Salud Ment (Mex). 2005;28:57–63.

Hojat M. Empathy in Patient Care: antecedents, Development, Measurement, and outcomes. New York: Springer; 2007.

Hojat M, Shannon SC, DeSantis J, Speicher MR, Bragan L, Calabrese LH. Does empathy decline in the clinical phase of medical education? A nationwide, multi-institutional, cross-sectional study of students at DO-granting medical schools. Acad Med 2020;95(6):911–918. https://doi.org/10.1097/ACM.0000000000003175 , PMID: 31977341.

Hojat M, Vergare MJ, Maxwell K et al. The devil is in the third year: a longitudinal study of erosion of empathy in medical school. Acad Med 2009;84(9):1182–1191. https://doi.org/10.1097/ACM.0b013e3181b17e55 . Erratum in: Acad Med 2009;84(9):1182–1191. PMID: 19707055.

Hojat M. Change in empathy in medical school. Med Educ 2018;52(4):456–457. https://doi.org/10.1111/medu.13497 , PMID: 29574956.

Magalhães E, Salgueira AP, Costa P, Costa MJ. Empathy in senior year and first year medical students: a cross-sectional study. B MC Med Educ 2011;11:52. https://doi.org/10.1186/1472-6920-11-52 , PMID: 21801365.

Ye X, Guo H, Xu Z, et al. Empathy variation of undergraduate medical students after early clinical contact: a cross-sectional study in China. BMJ Open. 2020;10:e035690. https://doi.org/10.1136/bmjopen-2019-035690 .

Assing Hvidt E, Søndergaard J, Wehberg S, Hvidt NC, Andersen CM. A cross-sectional study of student empathy across four medical schools in Denmark-associations between empathy level and age, sex, specialty preferences and motivation. BMC Med Educ 2022;22(1):489. https://doi.org/10.1186/s12909-022-03532-2 , PMID: 35739548.

Yu CC, Tan L, Le MK, et al. The development of empathy in the healthcare setting: a qualitative approach. BMC Med Educ. 2022;22(1):245. https://doi.org/10.1186/s12909-022-03312-y . PMID: 35379249, PMCID: PMC8981670.

Wanqi F. Chen panorama analysis and countermeasures of common nurse patient disputes in pediatrics. Qilu J Nurs. 2011;17(21):117–8. https://doi.org/10.3969/j.issn.1006-7256.2011.21.087 .

Hojat M, Gonnella JS, Nasca TJ, Mangione S, Vergare M, Magee M. Physician empathy: definition, components, measurement, and relationship to gender and specialty. Am J Psychiatry. 2002;159(9):1563–9.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Houghton C, Murphy K, Meehan B, Thomas J, Brooker D, Casey D. From screening to synthesis: using NVivo to enhance transparency in qualitative evidence synthesis. J Clin Nurs. 2017;26(5–6):873–81. https://doi.org/10.1111/jocn.13443 .

Hojat M, Gonnella JS, Mangione S et al. Empathy in medical students as related to academic performance, clinical competence and gender. Med Educ 2002;36(6):522–527. https://doi.org/10.1046/j.1365-2923.2002.01234.x , PMID: 12047665.

Zhao JB, Liang SW, Hou YF. Etc the relationship between empathy fatigue and post-traumatic stress disorder among clinical doctors. Guangdong Med. 2017;38(24):3841–4. https://doi.org/10.3969/j.issn.1001-9448.2017.24.036 .

Xu W, Zhang SC. Chinese pediatricians face a crisis: should they stay or leave? Pediatrics. 2014;134(6):1045–7. https://doi.org/10.1542/peds.2014-1377 . Epub 2014 Nov 10. PMID: 25384495.

Zhang X, Li L, Zhang Q, Le LH, Wu Y. Physician empathy in doctor-patient communication: A systematic review. Health Commun . 2023:1–11. doi: 10.1080/10410236.2023.2201735. Epub ahead of print. PMID: 37062918.

Libo J. Analysis and resolution of doctor-patient disputes in grassroots pediatric outpatient clinics. Chin Community Phys. 2020;36(10):184–.

Lingyan G. Research on Building a Good Pediatric Doctor-Patient Relationship from the perspective of Public Management [D]. Suzhou University; 2022. https://doi.org/10.27351/dcnki.gszhu.2022.000265 .

Wuying C, Lianqi L. The influence of context on empathy. Prog Psychol Sci. 2016;24(1):91–100. https://doi.org/10.3724/SP.J.1042.2016.00091 .

Song C, Du XT, Hong YX, Mao JH, Zhang W. Association between social supports and negative emotions among pediatric residents in China: the chain-mediating role of psychological resilience and burnout. Front Public Health 2022;10:962259. https://doi.org/10.3389/fpubh.2022.962259 , PMID: 36755738.

Jiaman S, Lihua L, Linling Y, et al. An analysis of the current situation and difficulties in informing cancer bad news. Chin Med Ethics. 2023;36(05):540–7.

Lases LSS, Arah OA, Busch ORC, Heineman MJ, Lombarts KMJMH. Learning climate positively influences residents’ work-related well-being. Adv Health Sci Educ Theory Pract 2019;24(2):317–330. https://doi.org/10.1007/s10459-018-9868-4 . Epub 2018 Dec 5. PMID: 30519786.

Min C, Zhen H, Mengxian L. Etc a survey and analysis of the effectiveness of cultivating empathy skills among medical students. Med Educ Res Pract. 2023;31(5):583–6. https://doi.org/10.13555/j.cnki.c.m.e.2023.05.014 .

Wei W, Ruiling Z, Jiongfeng Z. etc A literature review on the reasons and countermeasures for the shortage of pediatricians in China. Chizi , 2019 (15): 273.

Xinxin Y. A study on the factors influencing the vocational inclination and turnover behavior of Pediatric doctors in Public Medical institutions [D]. Guangdong: Southern Medical University; 2020.

Lei F, Chao S, Yunxia H. Etc analysis of the current situation and countermeasures of occupational burnout among standardized training students for pediatric resident physicians. Chin J Med Educ Explor. 2023;22(5):796–800. https://doi.org/10.3760/cma.j.cn116021-20220518-01297 .

Claramita M, Dalen JV, Van Der Vleuten CP. Doctors in a southeast Asian country communicate sub-optimally regardless of patients’ educational background. Patient Educ Couns. 2011;85(3):e169–74. https://doi.org/10.1016/j.pec.2011.02.002 . Epub 2011 Mar 21. PMID: 21420821.

Lim BT, Moriarty H, Huthwaite M, Gray L, Pullon S, Gallagher P. How well do medical students rate and communicate clinical empathy? Med Teach 2013;35(2):e946-e951. https://doi.org/10.3109/0142159X.2012.715783 . Epub 2012 Sep 3. PMID: 22938688.

Lim BT, Moriarty H, Huthwaite M. Being-in-role: A teaching innovation to enhance empathic communication skills in medical students. Med Teach 2011;33(12):e663-e669. https://doi.org/10.3109/0142159X.2011.611193 , PMID: 22225448.

Zhao J, Xiantao O, Li Q et al. Role of narrative medicine-based education in cultivating empathy in residents. BMC Med Educ 2023;23(1):124. https://doi.org/10.1186/s12909-023-04096-5 , PMID: 36810009.

Ziółkowska-Rudowicz E, Kładna A. Empathy-building of physicians. Part IV–development of skills enhancing capacity for empathy. Pol Merkur Lekarski. 2010;29(174):400–4. Polish. PMID: 21298994.

Haiyan G, Qinmei Z. Yongfei Z, etc the Impact of Bahrain Group activities on Empathy and Communication skills in standardized training of traditional Chinese medicine residents. J Traditional Chin Med Manage. 2019;27(12):103–5.

Download references

Acknowledgements

This work was financed by Postgraduate Medical Education Project in 2022 (BYH20220412); The 2022 Science and Technology Innovation Project (Humanities and Social Sciences) Project of Shanghai Jiao Tong University School of Medicine (WK2217); Fujian Medical University Education Reform Project: Application Research on the Intelligent Teaching Platform for Clinical Teachers under the Background of “New Medical Science” (J22021).

Author information

Pingping Li and Ling Weng contributed equally to this work and should be considered co-first authors.

Authors and Affiliations

Department of Pediatric Clinical Medicine School, Shanghai Children’s Medical Center, School of Medicine, Shanghai Jiao Tong University, Shanghai, 200127, China

Pingping Li & Lu Dong

Department of Science and Education, Fujian Maternity and Child Health Hospital, Fujian, 350000, China

You can also search for this author in PubMed   Google Scholar

Contributions

L.P.P. conceptualized the idea of this study. L.P.P. and W.L. contributed to design of the project and survey preparation and dissemination. L.P.P. contributed to investigate. D.L. contributed to writing-review and agreed to be accountable for all aspects of the work. All authors reviewed the manuscript.

Corresponding author

Correspondence to Lu Dong .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethics approval and consent to participate

Ethical approval for this study was obtained from the institutional research ethics committee of Shanghai Children’s Medical Center Affiliated to Shanghai Jiao Tong University School of Medicine(NO: SCMCTRB-K2023147-1). All participants received written explanations about the study in advance and signed a written consent form to participate.

Consent for publication

Not applicable.

None declared.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Li, P., Weng, L. & Dong, L. Empathy ability and influencing factors among pediatric residents in China: a mixed-methods study. BMC Med Educ 24 , 955 (2024). https://doi.org/10.1186/s12909-024-05858-5

Download citation

Received : 29 April 2024

Accepted : 02 August 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s12909-024-05858-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Empathy ability
• Pediatric residents

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Mixed reality in clinical settings for pediatric patients and their families: a literature review.

1. Introduction

1.1. importance of new technologies in health and education, 1.2. definitions and distinctions between ar, vr, and mr, 1.3. conceptual challenges and frameworks, 1.4. technological advancements and applications, 1.5. the role of ar and mr in modern healthcare and education, 1.6. psychological and physical risks of ar and vr, 1.7. gaps in research and the need for focused mr interventions, 2. materials and methods, 2.1. identification, 2.2. screening and eligibility, 2.3. included articles, 3.1. overview of included studies, 3.1.1. search results, 3.1.2. study characteristics, 3.2. general characteristics of the studies, 3.2.1. methods, 3.2.2. study population, 3.3. description of the interventions, 3.3.1. clinical settings of the mr application, 3.3.2. mr devices and software, 3.3.3. mr-based interventions.

• Early Adoption in Pain Management: Mott et al. [ 43 ], in 2008, represented the early adoption of MR in pediatric care. A child could visualize a 3D animation character from multiple angles in an MR system with audio narration, which made the child perform tasks. This highlights MR’s application in pain management and its ability to improve patient experiences in medical settings.
• Developmental and Educational Tools: By 2015, Bai et al. [ 38 ] established a setting for patients playing with augmented toys in a mirror MR display to improve and learn how to pretend to play, a crucial skill for autistic children. This showed the application of AR as a tool for social and cognitive development in children with ASD, promoting interaction and engagement. Similarly, a study by Calle-Bustos et al. [ 36 ] revealed MR’s role in interactive education for chronic health conditions, such as diabetes mellitus. An Android device overlaying MR food on a real dish was used in an AR game to support therapeutic education for children with diabetes mellitus, significantly contributing to self-management education in pediatric diabetes.
• Anxiety Reduction and Advanced MR Applications: In 2020, studies by Tait et al. [ 34 ] and Bray et al. [ 35 ] exemplified the maturity of MR technologies. Tait et al. used a printed storybook and an MR-enabled iPad program overlaying MR graphics, animations, and a chatbot with embedded interactive quizzes for information evaluation. Bray et al. employed the preloaded iPad Xploro ® , a digital therapeutic (DTx) platform that adopts AR, gameplay, and artificial intelligence, providing information on health environments, key health staff, and hospital equipment. Another study by Libaw et al. [ 37 ] in 2020 furthered the application of MR in clinical settings by applying the AR “Jenny the Robot” distraction technique during mask induction to encourage patients to take deep breaths.
• Social and Communication Skills Enhancement: Extracted from the primary study of Karami et al. [ 32 ], five individual studies on ASD conducted between 2015 and 2018 applied MR interventions for various social and communication skill-enhancing purposes. In a study by Chen et al. [ 39 ], AR-based Video-Modeling with Storybook (ARVMS)—comprising seven sessions—was devised to learn the facial expressions and emotions of others in social situations.
• Integration with Therapeutic Methods: Other studies targeting children with ASD, such as those by Kurniawan [ 40 ] and Nubia et al. [ 41 ], showed a continued trend toward using MR for enhancing communication and social skills. Kurniawan developed the Picture Exchange Communication System with AR-based multimedia using visual aids in a more interactive format to improve the communication abilities of children with ASD. In contrast, Nubia et al. introduced the process and simulation of an MR-based pictogram recognition task to improve the attention process and the appearance of verbal language in participants with ASD.
• Innovative Smart Glasses Applications: Vahabzadeh et al. [ 42 ] illustrated the innovative use of smart glasses-based interventions. Specifically, they reported that Empowered Brain, a smart glasses-based social communication and behavioral intervention, was used to improve the duration of gaze at faces and reduce ADHD symptoms in children, adolescents, and young adults with ASD.

3.4. Outcomes

3.4.1. variables, 3.4.2. main effect, 4. discussion, 4.1. study design and methodological considerations, 4.2. intervention characteristics and technological implementation, 4.3. population characteristics and clinical settings, 4.4. functional outcomes and practical implications, 4.5. intervention settings, technological implementation, and functional outcomes, 4.6. interrelationships between functional outcomes, 4.7. family involvement, care continuity, and future research directions, 4.8. limitations and future research directions, 5. conclusions, conflicts of interest.

• Barteit, S.; Lanfermann, L.; Bärnighausen, T.; Neuhann, F.; Beiersmann, C. Augmented, Mixed, and Virtual Reality-Based Head-Mounted Devices for Medical Education: Systematic Review. JMIR Serious Games 2021 , 9 , e29080. [ Google Scholar ] [ CrossRef ]
• Nesenbergs, K.; Abolins, V.; Ormanis, J.; Mednis, A. Use of Augmented and Virtual Reality in Remote Higher Education: A Systematic Umbrella Review. Educ. Sci. 2021 , 11 , 8. [ Google Scholar ] [ CrossRef ]
• Kaplan, A.D.; Cruit, J.; Endsley, M.; Beers, S.M.; Sawyer, B.D.; Hancock, P.A. The Effects of Virtual Reality, Augmented Reality, and Mixed Reality as Training Enhancement Methods: A Meta-Analysis. Hum. Factors 2021 , 63 , 706–726. [ Google Scholar ] [ CrossRef ]
• Sala, N. Virtual Reality, Augmented Reality, and Mixed Reality in Education: A Brief Overview. In Current and Prospective Applications of Virtual Reality in Higher Education ; Choi, D., Dailey-Hebert, A., Estes, J., Eds.; IGI Global: Hershey, PA, USA, 2021; pp. 48–73. [ Google Scholar ] [ CrossRef ]
• Gaol, F.L.; Prasolova-Førland, E. Special Section Editorial: The Frontiers of Augmented and Mixed Reality in All Levels of Education. Educ. Inf. Technol. 2022 , 27 , 611–623. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Rokhsaritalemi, S.; Sadeghi-Niaraki, A.; Choi, S.M. A Review on Mixed Reality: Current Trends, Challenges and Prospects. Appl. Sci. 2020 , 10 , 636. [ Google Scholar ] [ CrossRef ]
• Rauschnabel, P.A.; Felix, R.; Hinsch, C.; Shahab, H.; Alt, F. What Is XR? Towards a Framework for Augmented and Virtual Reality. Comput. Hum. Behav. 2022 , 133 , 107289. [ Google Scholar ] [ CrossRef ]
• Xi, N.; Chen, J.; Gama, F.; Riar, M.; Hamari, J. The Challenges of Entering the Metaverse: An Experiment on the Effect of Extended Reality on Workload. Inf. Syst. Front. 2023 , 25 , 659–680. [ Google Scholar ] [ CrossRef ]
• Speicher, M.; Hall, B.D.; Nebeling, M. What Is Mixed Reality? In Proceedings of the CHI Conference on Human Factors in Computing Systems, Glasgow, UK, 4–9 May 2019; ACM: New York, NY, USA, 2019; pp. 1–15. [ Google Scholar ] [ CrossRef ]
• Solbiati, L.; Gennaro, N.; Muglia, R. Augmented Reality: From Video Games to Medical Clinical Practice. Cardiovasc. Interv. Radiol. 2020 , 43 , 1427–1429. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gupta, R.; He, J.; Ranjan, R.; Gan, W.-S.; Klein, F.; Schneiderwind, C.; Neidhardt, A.; Brandenburg, K.; Valimaki, V. Augmented/Mixed Reality Audio for Hearables: Sensing, Control, and Rendering. IEEE Signal Process. Mag. 2022 , 39 , 63–89. [ Google Scholar ] [ CrossRef ]
• Nasr, R.S.; El-Deeb, S. Exploring Mixed Reality: Enhancing Consumer Interaction. In Advances in Marketing, Customer Relationship Management, and E-Services ; Pires, P.B., Santos, J.D., Pereira, I.V., Torres, A.I., Eds.; IGI Global: Hershey, PA, USA, 2023; pp. 234–251. [ Google Scholar ] [ CrossRef ]
• Tene, T.; Vique López, D.F.; Valverde Aguirre, P.E.; Orna Puente, L.M.; Vacacela Gomez, C. Virtual Reality and Augmented Reality in Medical Education: An Umbrella Review. Front. Digit. Health 2024 , 6 , 1365345. [ Google Scholar ] [ CrossRef ]
• Xu, X.; Mangina, E.; Campbell, A.G. HMD-Based Virtual and Augmented Reality in Medical Education: A Systematic Review. Front. Virtual Real. 2021 , 2 , 69210. [ Google Scholar ] [ CrossRef ]
• Wu, Y.; Hu, K.; Chen, D.Z.; Wu, J. AI-Enhanced Virtual Reality in Medicine: A Comprehensive Survey. In Proceedings of the International Joint Conference on Artificial Intelligence, Jeju, Republic of Korea, 3–9 August 2024; p. 920. [ Google Scholar ] [ CrossRef ]
• Mergen, M.; Graf, N.; Meyerheim, M. Reviewing the Current State of Virtual Reality Integration in Medical Education—A Scoping Review. BMC Med. Educ. 2024 , 24 , 788. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ro, J.; Hong, C.W. Technology Trends of Virtual Augmented Reality and Application to Military Education and Training. Inst. Inf. Technol. 2022 , 20 , 151–164. [ Google Scholar ] [ CrossRef ]
• Syahry, A.; Basuki, D.K.; Sukaridhoto, S.; Budiarti, R.P.N.; Hanifati, K.; Muntahir, I. Implementation of Augmented Reality in Medical Education. In Proceedings of the 2022 International Electronics Symposium (IES), Surabaya, Indonesia, 9–11 August 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 684–690. [ Google Scholar ] [ CrossRef ]
• Kim, M.; Choi, S.H.; Park, K.B.; Lee, J.Y. User Interactions for Augmented Reality Smart Glasses: A Comparative Evaluation of Visual Contexts and Interaction Gestures. Appl. Sci. 2019 , 9 , 3171. [ Google Scholar ] [ CrossRef ]
• Chengoden, R.; Victor, N.; Huynh-The, T.; Yenduri, G.; Jhaveri, R.H.; Alazab, M.; Bhattacharya, S.; Hegde, P.; Maddikunta, P.K.R.; Gadekallu, T.R. Metaverse for Healthcare: A Survey on Potential Applications, Challenges, and Future Directions. IEEE Access 2023 , 11 , 12765–12795. [ Google Scholar ] [ CrossRef ]
• Moro, C.; Phelps, C.; Redmond, P.; Stromberga, Z. HoloLens and Mobile Augmented Reality in Medical and Health Science Education: A Randomised Controlled Trial. Br. J. Educ. Technol. 2021 , 52 , 680–694. [ Google Scholar ] [ CrossRef ]
• Han, S.; Lim, C. Research Trends on Augmented Reality Education in Korea from 2008 to 2019. J. Educ. Technol. 2020 , 36 , 505–528. [ Google Scholar ] [ CrossRef ]
• Nam, S.H.; Lee, J.M. Augmented Reality in Education: A Meta-Analysis. J. Educ. Media 2020 , 26 , 129–156. [ Google Scholar ] [ CrossRef ]
• Yoo, M.; Kim, J.; Koo, Y.; Song, J.H. A Meta-Analysis on Effects of VR, AR, MR-Based Learning in Korea. J. Educ. Media 2018 , 24 , 459–488. [ Google Scholar ] [ CrossRef ]
• Dhar, P.; Rocks, T.; Samarasinghe, R.M.; Stephenson, G.; Smith, C. Augmented Reality in Medical Education: Students’ Experiences and Learning Outcomes. Med. Educ. Online 2021 , 26 , 1953953. [ Google Scholar ] [ CrossRef ]
• O’Shiel, D. Disappearing Boundaries? Reality, Virtuality, and the Possibility of “Pure” Mixed Reality (MR). Indo-Pac. J. Phenomenol. 2020 , 20 , e1887570. [ Google Scholar ] [ CrossRef ]
• Kim, S.; Ryu, J.; Choi, Y.; Kang, Y.; Li, H.; Kim, K. Eye-Contact Game Using Mixed Reality for the Treatment of Children with Attention Deficit Hyperactivity Disorder. IEEE Access 2020 , 8 , 45996–46006. [ Google Scholar ] [ CrossRef ]
• Aromataris, E.; Fernandez, R.; Godfrey, C.M.; Holly, C.; Khalil, H.; Tungpunkom, P. Summarizing Systematic Reviews: Methodological Development, Conduct, and Reporting of an Umbrella Review Approach. Int. J. Evid. Based Healthc. 2015 , 13 , 132–340. [ Google Scholar ] [ CrossRef ]
• Page, M.J.; Moher, D.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. PRISMA 2020 Explanation and Elaboration: Updated Guidance and Exemplars for Reporting Systematic Reviews. BMJ 2021 , 372 , n160. [ Google Scholar ] [ CrossRef ]
• Urlings, J.; Sezer, S.; Ter Laan, M.; Bartels, R.; Maal, T.; Boogaarts, J.; Henssen, D. The Role and Effectiveness of Augmented Reality in Patient Education: A Systematic Review of the Literature. Patient Educ. Couns. 2022 , 105 , 1917–1927. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Alqudimat, M.; Mesaroli, G.; Lalloo, C.; Stinson, J.; Matava, C. State of the Art: Immersive Technologies for Perioperative Anxiety, Acute, and Chronic Pain Management in Pediatric Patients. Curr. Anesthesiol. Rep. 2021 , 11 , 265–274. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Karami, B.; Koushki, R.; Arabgol, F.; Rahmani, M.; Vahabie, A.H. Effectiveness of Virtual/Augmented Reality–Based Therapeutic Interventions on Individuals with Autism Spectrum Disorder: A Comprehensive Meta-Analysis. Front. Psychiatry 2021 , 12 , 665326. [ Google Scholar ] [ CrossRef ]
• Gasteratos, K.; Papakonstantinou, M.; Man, A.; Babatsikos, E.; Tamalonis, A.; Goverman, J. Adjunctive Nonpharmacologic Interventions for the Management of Burn Pain: A Systematic Review. Plast. Reconstr. Surg. 2022 , 149 , 985e–994e. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tait, A.R.; Connally, L.; Doshi, A.; Johnson, A.; Skrzpek, A.; Grimes, M.; Becher, A.; Choi, J.E.; Weber, M. Development and Evaluation of an Augmented Reality Education Program for Pediatric Research. J. Clin. Transl. Res. 2020 , 5 , 96–101. [ Google Scholar ]
• Bray, L.; Sharpe, A.; Gichuru, P.; Fortune, P.M.; Blake, L.; Appleton, V. The Acceptability and Impact of the Xploro Digital Therapeutic Platform to Inform and Prepare Children for Planned Procedures in a Hospital: Before and After Evaluation Study. J. Med. Internet Res. 2020 , 22 , e17367. [ Google Scholar ] [ CrossRef ]
• Calle-Bustos, A.M.; Juan, M.C.; García-García, I.; Abad, F. An Augmented Reality Game to Support Therapeutic Education for Children with Diabetes. PLoS ONE 2017 , 12 , e0184645. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Libaw, J.S.; Sinskey, J.L. Use of Augmented Reality During Inhaled Induction of General Anesthesia in 3 Pediatric Patients: A Case Report. A A Pract. 2020 , 14 , e01219. [ Google Scholar ] [ CrossRef ]
• Bai, Z.; Blackwell, A.F.; Coulouris, G. Using Augmented Reality to Elicit Pretend Play for Children with Autism. IEEE Trans. Vis. Comput. Graph. 2015 , 21 , 598–610. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Chen, C.-H.; Lee, I.-J.; Lin, L.-Y. Augmented Reality-Based Video-Modeling Storybook of Non-Verbal Facial Cues for Children with Autism Spectrum Disorder to Improve Their Perceptions and Judgments of Facial Expressions and Emotions. Comput. Hum. Behav. 2016 , 55 , 477–485. [ Google Scholar ] [ CrossRef ]
• Taryadi; Kurniawan, I. The Improvement of Autism Spectrum Disorders on Children Communication Ability with PECS Method Multimedia Augmented Reality-Based. J. Phys. Conf. Ser. 2018 , 2018 , 12009. [ Google Scholar ] [ CrossRef ]
• Nubia, R.M.; Fabián, G.R.; Wilson, R.A.; Wilmer, P.B. Development of a Mobile Application in Augmented Reality to Improve the Communication Field of Autistic Children at a Neurorehabilitar Clinic. In Proceedings of the 2015 Workshop on Engineering Applications—International Congress on Engineering (WEA), Bogota, Colombia, 1–4 July 2015; pp. 1–6. [ Google Scholar ] [ CrossRef ]
• Vahabzadeh, A.; Keshav, N.U.; Salisbury, J.P.; Sahin, N.T. Improvement of Attention-Deficit/Hyperactivity Disorder Symptoms in School-Aged Children, Adolescents, and Young Adults with Autism via a Digital Smartglasses-Based Socioemotional Coaching Aid: Short-Term, Uncontrolled Pilot Study. JMIR Ment. Health 2018 , 5 , e25. [ Google Scholar ] [ CrossRef ]
• Mott, J.; Bucolo, S.; Cuttle, L.; Mill, J.; Hilder, M.; Miller, K.; Kimble, R.M. The Efficacy of an Augmented Virtual Reality System to Alleviate Pain in Children Undergoing Burns Dressing Changes: A Randomised Controlled Trial. Burns 2008 , 34 , 803–808. [ Google Scholar ] [ CrossRef ]
• Cao, J.; Lam, K.Y.; Lee, L.H.; Liu, X.; Hui, P.; Su, X. Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence. ACM Comput. Surv. 2023 , 55 , 1–36. [ Google Scholar ] [ CrossRef ]
• Dargan, S.; Bansal, S.; Kumar, M.; Mittal, A.; Kumar, K. Augmented Reality: A Comprehensive Review. Arch. Comput. Methods Eng. 2023 , 30 , 1057–1080. [ Google Scholar ] [ CrossRef ]
• Wu, B.; Yu, X.; Gu, X. Effectiveness of Immersive Virtual Reality Using Head-Mounted Displays on Learning Performance: A Meta-Analysis. Br. J. Educ. Technol. 2020 , 51 , 1991–2005. [ Google Scholar ] [ CrossRef ]
• Xing, Y.; Liang, Z.; Fahy, C.; Shell, J.; Guan, K.; Liu, Y.; Zhang, Q. Virtual Reality Research: Design Virtual Education System for Epidemic (COVID-19) Knowledge to Public. Appl. Sci. 2021 , 11 , 10586. [ Google Scholar ] [ CrossRef ]
• Lovreglio, R.; Duan, X.; Rahouti, A.; Phipps, R.; Nilsson, D. Comparing the Effectiveness of Fire Extinguisher Virtual Reality and Video Training. Virtual Real. 2020 , 25 , 133–145. [ Google Scholar ] [ CrossRef ]
• Schmidli, H.; Häring, D.A.; Thomas, M.; Cassidy, A.; Weber, S.; Bretz, F. Beyond Randomized Clinical Trials: Use of External Controls. Clin. Pharmacol. Therap. 2020 , 107 , 806–816. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Levin, J.R.; Ferron, J.M.; Gafurov, B.S. Investigation of Single-Case Multiple-Baseline Randomization Tests of Trend and Variability. Educ. Psychol. Rev. 2021 , 33 , 713–737. [ Google Scholar ] [ CrossRef ]
• Beling, C.; Skinner, B.; Bluff, A. Xploro: Multi-User AR. In Proceedings of the ACM SIGGRAPH 2019 Appy Hour, Los Angeles, CA, USA, 28 July–1 August 2019; ACM: New York, NY, USA, 2019; pp. 1–2. [ Google Scholar ] [ CrossRef ]
• Sahin, N.T.; Keshav, N.U.; Salisbury, J.P.; Vahabzadeh, A. Safety and Lack of Negative Effects of Wearable Augmented-Reality Social Communication Aid for Children and Adults with Autism. J. Clin. Med. 2018 , 7 , 188. [ Google Scholar ] [ CrossRef ]
• Law, E.L.-C.; Heintz, M. Augmented Reality Applications for K-12 Education: A Systematic Review from the Usability and User Experience Perspective. Int. J. Child Comput. Interact. 2021 , 30 , 100321. [ Google Scholar ] [ CrossRef ]
• Ozdemir, M.; Sahin, C.; Arcagok, S.; Demir, M.K. The Effect of Augmented Reality Applications in the Learning Process: A Meta-Analysis Study. Eurasian J. Educ. Res. 2018 , 18 , 165–186. [ Google Scholar ] [ CrossRef ]
• Slater, M.; Gonzalez-Liencres, C.; Haggard, P.; Vinkers, C.; Gregory-Clarke, R.; Jelley, S.; Watson, Z.; Breen, G.; Schwarz, R.; Steptoe, W.; et al. The Ethics of Realism in Virtual and Augmented Reality. Front. Virtual Real. 2020 , 1 , 512449. [ Google Scholar ] [ CrossRef ]
• Csikszentmihalyi, M.; Csikszentmihalyi, M. Toward a Psychology of Optimal Experience. In Flow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi ; Csikszentmihalyi, M., Ed.; Springer: Dordrecht, The Netherlands, 2014; pp. 209–226. [ Google Scholar ] [ CrossRef ]
• Novak, T.P.; Hoffman, D.L.; Yung, Y.-F. Measuring the Customer Experience in Online Environments: A Structural Modeling Approach. Mark. Sci. 2000 , 19 , 22–42. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Sin, J.E.; Kim, A.R. Mixed Reality in Clinical Settings for Pediatric Patients and Their Families: A Literature Review. Int. J. Environ. Res. Public Health 2024 , 21 , 1185. https://doi.org/10.3390/ijerph21091185

Sin JE, Kim AR. Mixed Reality in Clinical Settings for Pediatric Patients and Their Families: A Literature Review. International Journal of Environmental Research and Public Health . 2024; 21(9):1185. https://doi.org/10.3390/ijerph21091185

Sin, Jae Eun, and Ah Rim Kim. 2024. "Mixed Reality in Clinical Settings for Pediatric Patients and Their Families: A Literature Review" International Journal of Environmental Research and Public Health 21, no. 9: 1185. https://doi.org/10.3390/ijerph21091185

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals