what is quantitative research in public health

An official website of the United States government

Here's how you know

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Common Data Types in Public Health Research

Quantitative data.

• Quantitative data is measurable, often used for comparisons, and involves counting of people, behaviors, conditions, or other discrete events (Wang, 2013).
• Quantitative data uses numbers to determine the what, who, when, and where of health-related events (Wang, 2013).
• Examples of quantitative data include: age, weight, temperature, or the number of people suffering from diabetes.

Qualitative Data

• Qualitative data is a broad category of data that can include almost any non-numerical data.
• Qualitative data uses words to describe a particular health-related event (Romano).
• This data can be observed, but not measured.
• Involves observing people in selected places and listening to discover how they feel and why they might feel that way (Wang, 2013).
• Examples of qualitative data include: male/female, smoker/non-smoker, or questionnaire response (agree, disagree, neutral).
• Measuring organizational change.
• Measures of clinical leadership in implementing evidence-based guidelines.
• Patient perceptions of quality of care.

Data Sources

Primary data sources.

• Primary data analysis in which the same individual or team of researchers designs, collects, and analyzes the data, for the purpose of answering a research question (Koziol & Arthur, nd).

Advantages to Using Primary Data

• You collect exactly the data elements that you need to answer your research question (Romano).
• You can test an intervention, such as an experimental drug or an educational program, in the purest way (a double-blind randomized controlled trial (Romano).
• You control the data collection process, so you can ensure data quality, minimize the number of missing values, and assess the reliability of your instruments (Romano).

Secondary Data Sources

• Existing data collected for another purposes, that you use to answer your research question (Romano).

Advantages of Working with Secondary Data

• Large samples
• Can provide population estimates : for example state data can be combined across states to get national estimates (Shaheen, Pan, & Mukherjee).
• Less expensive to collect than primary data (Romano)
• It takes less time to collect secondary data (Romano).
• You may not need to worry about informed consent, human subjects restriction (Romano).

Issues in Using Secondary Data

• Study design and data collection already completed (Koziol & Arthur, nd).
• Data may not facilitate particular research question o Information regarding study design and data collection procedures may be scarce.
• Data may potentially lack depth (the greater the breadth the harder it is to measure any one construct in depth) (Koziol & Arthur, nd).
• Certain fields or departments (e.g., experimental programs) may place less value on secondary data analysis (Koziol & Arthur, nd).
• Often requires special techniques to analyze statistically the data.

• Research article
• Open access
• Published: 03 February 2021

A review of the quantitative effectiveness evidence synthesis methods used in public health intervention guidelines

• Ellesha A. Smith   ORCID: orcid.org/0000-0002-4241-7205 1 ,
• Nicola J. Cooper 1 ,
• Alex J. Sutton 1 ,
• Keith R. Abrams 1 &
• Stephanie J. Hubbard 1  

BMC Public Health volume  21 , Article number:  278 ( 2021 ) Cite this article

3908 Accesses

5 Citations

3 Altmetric

Metrics details

The complexity of public health interventions create challenges in evaluating their effectiveness. There have been huge advancements in quantitative evidence synthesis methods development (including meta-analysis) for dealing with heterogeneity of intervention effects, inappropriate ‘lumping’ of interventions, adjusting for different populations and outcomes and the inclusion of various study types. Growing awareness of the importance of using all available evidence has led to the publication of guidance documents for implementing methods to improve decision making by answering policy relevant questions.

The first part of this paper reviews the methods used to synthesise quantitative effectiveness evidence in public health guidelines by the National Institute for Health and Care Excellence (NICE) that had been published or updated since the previous review in 2012 until the 19th August 2019.The second part of this paper provides an update of the statistical methods and explains how they address issues related to evaluating effectiveness evidence of public health interventions.

The proportion of NICE public health guidelines that used a meta-analysis as part of the synthesis of effectiveness evidence has increased since the previous review in 2012 from 23% (9 out of 39) to 31% (14 out of 45). The proportion of NICE guidelines that synthesised the evidence using only a narrative review decreased from 74% (29 out of 39) to 60% (27 out of 45).An application in the prevention of accidents in children at home illustrated how the choice of synthesis methods can enable more informed decision making by defining and estimating the effectiveness of more distinct interventions, including combinations of intervention components, and identifying subgroups in which interventions are most effective.

Conclusions

Despite methodology development and the publication of guidance documents to address issues in public health intervention evaluation since the original review, NICE public health guidelines are not making full use of meta-analysis and other tools that would provide decision makers with fuller information with which to develop policy. There is an evident need to facilitate the translation of the synthesis methods into a public health context and encourage the use of methods to improve decision making.

Peer Review reports

To make well-informed decisions and provide the best guidance in health care policy, it is essential to have a clear framework for synthesising good quality evidence on the effectiveness and cost-effectiveness of health interventions. There is a broad range of methods available for evidence synthesis. Narrative reviews provide a qualitative summary of the effectiveness of the interventions. Meta-analysis is a statistical method that pools evidence from multiple independent sources [ 1 ]. Meta-analysis and more complex variations of meta-analysis have been extensively applied in the appraisals of clinical interventions and treatments, such as drugs, as the interventions and populations are clearly defined and tested in randomised, controlled conditions. In comparison, public health studies are often more complex in design, making synthesis more challenging [ 2 ].

Many challenges are faced in the synthesis of public health interventions. There is often increased methodological heterogeneity due to the inclusion of different study designs. Interventions are often poorly described in the literature which may result in variation within the intervention groups. There can be a wide range of outcomes, whose definitions are not consistent across studies. Intermediate, or surrogate, outcomes are often used in studies evaluating public health interventions [ 3 ]. In addition to these challenges, public health interventions are often also complex meaning that they are made up of multiple, interacting components [ 4 ]. Recent guidance documents have focused on the synthesis of complex interventions [ 2 , 5 , 6 ]. The National Institute for Health and Care Excellence (NICE) guidance manual provides recommendations across all topics that are covered by NICE and there is currently no guidance that focuses specifically on the public health context.

Research questions

A methodological review of NICE public health intervention guidelines by Achana et al. (2014) found that meta-analysis methods were not being used [ 3 ]. The first part of this paper aims to update and compare, to the original review, the meta-analysis methods being used in evidence synthesis of public health intervention appraisals.

The second part of this paper aims to illustrate what methods are available to address the challenges of public health intervention evidence synthesis. Synthesis methods that go beyond a pairwise meta-analysis are illustrated through the application to a case study in public health and are discussed to understand how evidence synthesis methods can enable more informed decision making.

The third part of this paper presents software, guidance documents and web tools for methods that aim to make appropriate evidence synthesis of public health interventions more accessible. Recommendations for future research and guidance production that can improve the uptake of these methods in a public health context are discussed.

Update of NICE public health intervention guidelines review

Nice guidelines.

The National Institute for Health and Care Excellence (NICE) was established in 1999 as a health authority to provide guidance on new medical technologies to the NHS in England and Wales [ 7 ]. Using an evidence-based approach, it provides recommendations based on effectiveness and cost-effectiveness to ensure an open and transparent process of allocating NHS resources [ 8 ]. The remit for NICE guideline production was extended to public health in April 2005 and the first recommendations were published in March 2006. NICE published ‘Developing NICE guidelines: the manual’ in 2006, which has been updated since, with the most recent in 2018 [ 9 ]. It was intended to be a guidance document to aid in the production of NICE guidelines across all NICE topics. In terms of synthesising quantitative evidence, the NICE recommendations state: ‘meta-analysis may be appropriate if treatment estimates of the same outcome from more than 1 study are available’ and ‘when multiple competing options are being appraised, a network meta-analysis should be considered’. The implementation of network meta-analysis (NMA), which is described later, as a recommendation from NICE was introduced into the guidance document in 2014, with a further update in 2018.

Background to the previous review

The paper by Achana et al. (2014) explored the use of evidence synthesis methodology in NICE public health intervention guidelines published between 2006 and 2012 [ 3 ]. The authors conducted a systematic review of the methods used to synthesise quantitative effectiveness evidence within NICE public health guidelines. They found that only 23% of NICE public health guidelines used pairwise meta-analysis as part of the effectiveness review and the remainder used a narrative summary or no synthesis of evidence at all. The authors argued that despite significant advances in the methodology of evidence synthesis, the uptake of methods in public health intervention evaluation is lower than other fields, including clinical treatment evaluation. The paper concluded that more sophisticated methods in evidence synthesis should be considered to aid in decision making in the public health context [ 3 ].

The search strategy used in this paper was equivalent to that in the previous paper by Achana et al. (2014)[ 3 ]. The search was conducted through the NICE website ( https://www.nice.org.uk/guidance ) by searching the ‘Guidance and Advice List’ and filtering by ‘Public Health Guidelines’ [ 10 ]. The search criteria included all guidance documents that had been published from inception (March 2006) until the 19th August 2019. Since the original review, many of the guidelines had been updated with new documents or merged. Guidelines that remained unchanged since the previous review in 2012 were excluded and used for comparison.

The guidelines contained multiple documents that were assessed for relevance. A systematic review is a separate synthesis within a guideline that systematically collates all evidence on a specific research question of interest in the literature. Systematic reviews of quantitative effectiveness, cost-effectiveness evidence and decision modelling reports were all included as relevant. Qualitative reviews, field reports, expert opinions, surveillance reports, review decisions and other supporting documents were excluded at the search stage.

Within the reports, data was extracted on the types of review (narrative summary, pairwise meta-analysis, network meta-analysis (NMA), cost-effectiveness review or decision model), design of included primary studies (randomised controlled trials or non-randomised studies, intermediate or final outcomes, description of outcomes, outcome measure statistic), details of the synthesis methods used in the effectiveness evaluation (type of synthesis, fixed or random effects model, study quality assessment, publication bias assessment, presentation of results, software). Further details of the interventions were also recorded, including whether multiple interventions were lumped together for a pairwise comparison, whether interventions were complex (made up of multiple components) and details of the components. The reports were also assessed for potential use of complex intervention evidence synthesis methodology, meaning that the interventions that were evaluated in the review were made up of components that could potentially be synthesised using an NMA or a component NMA [ 11 ]. Where meta-analysis was not used to synthesis effectiveness evidence, the reasons for this was also recorded.

Search results and types of reviews

There were 67 NICE public health guidelines available on the NICE website. A summary flow diagram describing the literature identification process and the list of guidelines and their reference codes are provided in Additional files  1 and 2 . Since the previous review, 22 guidelines had not been updated. The results from the previous review were used for comparison to the 45 guidelines that were either newly published or updated.

The guidelines consisted of 508 documents that were assessed for relevance. Table  1 shows which types of relevant documents were available in each of the 45 guidelines. The median number of relevant articles per guideline was 3 (minimum = 0, maximum = 10). Two (4%) of the NICE public health guidelines did not report any type of systematic review, cost-effectiveness review or decision model (NG68, NG64) that met the inclusion criteria. 167 documents from 43 NICE public health guidelines were systematic reviews of quantitative effectiveness, cost-effectiveness or decision model reports and met the inclusion criteria.

Narrative reviews of effectiveness were implemented in 41 (91%) of the NICE PH guidelines. 14 (31%) contained a review that used meta-analysis to synthesise the evidence. Only one (1%) NICE guideline contained a review that implemented NMA to synthesise the effectiveness of multiple interventions; this was the same guideline that used NMA in the original review and had been updated. 33 (73%) guidelines contained cost-effectiveness reviews and 34 (76%) developed a decision model.

Comparison of review types to original review

Table  2 compares the results of the update to the original review and shows that the types of reviews and evidence synthesis methodologies remain largely unchanged since 2012. The proportion of guidelines that only contain narrative reviews to synthesise effectiveness or cost-effectiveness evidence has reduced from 74% to 60% and the proportion that included a meta-analysis has increased from 23% to 31%. The proportion of guidelines with reviews that only included evidence from randomised controlled trials and assessed the quality of individual studies remained similar to the original review.

Characteristics of guidelines using meta-analytic methods

Table  3 details the characteristics of the meta-analytic methods implemented in 24 reviews of the 14 guidelines that included one. All of the reviews reported an assessment of study quality, 12 (50%) reviews included only data from randomised controlled trials, 4 (17%) reviews used intermediate outcomes (e.g. uptake of chlamydia screening rather than prevention of chlamydia (PH3)), compared to the 20 (83%) reviews that used final outcomes (e.g. smoking cessation rather than uptake of a smoking cessation programme (NG92)). 2 (8%) reviews only used a fixed effect meta-analysis, 19 (79%) reviews used a random effects meta-analysis and 3 (13%) did not report which they had used.

An evaluation of the intervention information reported in the reviews concluded that 12 (50%) reviews had lumped multiple (more than two) different interventions into a control versus intervention pairwise meta-analysis. Eleven (46%) of the reviews evaluated interventions that are made up of multiple components (e.g. interventions for preventing obesity in PH47 were made up of diet, physical activity and behavioural change components).

21 (88%) of the reviews presented the results of the meta-analysis in the form of a forest plot and 22 (92%) presented the results in the text of the report. 20 (83%) of the reviews used two or more forms of presentation for the results. Only three (13%) reviews assessed publication bias. The most common software to perform meta-analysis was RevMan in 14 (58%) of the reviews.

Reasons for not using meta-analytic methods

The 143 reviews of effectiveness and cost effectiveness that did not use meta-analysis methods to synthesise the quantitative effectiveness evidence were searched for reasons behind this decision. 70 reports (49%) did not give a reason for not synthesising the data using a meta-analysis and 164 reasons were reported which are displayed in Fig.  1 . Out of the remaining reviews, multiple reasons for not using a meta-analysis were given. 53 (37%) of the reviews reported at least one reason due to heterogeneity. 30 (21%) decision model reports did not give a reason and these are categorised separately. 5 (3%) reviews reported that meta-analysis was not applicable or feasible, 1 (1%) reported that they were following NICE guidelines and 5 (3%) reported that there were a lack of studies.

Frequency and proportions of reasons reported for not using statistical methods in quantitative evidence synthesis in NICE PH intervention reviews

The frequency of reviews and guidelines that used meta-analytic methods were plotted against year of publication, which is reported in Fig.  2 . This showed that the number of reviews that used meta-analysis were approximately constant but there is some suggestion that the number of meta-analyses used per guideline increased, particularly in 2018.

Number of meta-analyses in NICE PH guidelines by year. Guidelines that were published before 2012 had been updated since the previous review by Achana et al. (2014) [ 3 ]

Comparison of meta-analysis characteristics to original review

Table  4 compares the characteristics of the meta-analyses used in the evidence synthesis of NICE public health intervention guidelines to the original review by Achana et al. (2014) [ 3 ]. Overall, the characteristics in the updated review have not much changed from those in the original. These changes demonstrate that the use of meta-analysis in NICE guidelines has increased but remains low. Lumping of interventions still appears to be common in 50% of reviews. The implications of this are discussed in the next section.

Application of evidence synthesis methodology in a public health intervention: motivating example

Since the original review, evidence synthesis methods have been developed and can address some of the challenges of synthesising quantitative effectiveness evidence of public health interventions. Despite this, the previous section shows that the uptake of these methods is still low in NICE public health guidelines - usually limited to a pairwise meta-analysis.

It has been shown in the results above and elsewhere [ 12 ] that heterogeneity is a common reason for not synthesising the quantitative effectiveness evidence available from systematic reviews in public health. Statistical heterogeneity is the variation in the intervention effects between the individual studies. Heterogeneity is problematic in evidence synthesis as it leads to uncertainty in the pooled effect estimates in a meta-analysis which can make it difficult to interpret the pooled results and draw conclusions. Rather than exploring the source of the heterogeneity, often in public health intervention appraisals a random effects model is fitted which assumes that the study intervention effects are not equivalent but come from a common distribution [ 13 , 14 ]. Alternatively, as demonstrated in the review update, heterogeneity is used as a reason to not undertake any quantitative evidence synthesis at all.

Since the size of the intervention effects and the methodological variation in the studies will affect the impact of the heterogeneity on a meta-analysis, it is inappropriate to base the methodological approach of a review on the degree of heterogeneity, especially within public health intervention appraisal where heterogeneity seems inevitable. Ioannidis et al. (2008) argued that there are ‘almost always’ quantitative synthesis options that may offer some useful insights in the presence of heterogeneity, as long as the reviewers interpret the findings with respect to their limitations [ 12 ].

In this section current evidence synthesis methods are applied to a motivating example in public health. This aims to demonstrate that methods beyond pairwise meta-analysis can provide appropriate and pragmatic information to public health decision makers to enable more informed decision making.

Figure  3 summarises the narrative of this part of the paper and illustrates the methods that are discussed. The red boxes represent the challenges in synthesising quantitative effectiveness evidence and refers to the section within the paper for more detail. The blue boxes represent the methods that can be applied to investigate each challenge.

Summary of challenges that are faces in the evidence synthesis of public health interventions and methods that are discussed to overcome these challenges

Evaluating the effect of interventions for promoting the safe storage of cleaning products to prevent childhood poisoning accidents

To illustrate the methodological developments, a motivating example is used from the five year, NIHR funded, Keeping Children Safe Programme [ 15 ]. The project included a Cochrane systematic review that aimed to increase the use of safety equipment to prevent accidents at home in children under five years old. This application is intended to be illustrative of the benefits of new evidence synthesis methods since the previous review. It is not a complete, comprehensive analysis as it only uses a subset of the original dataset and therefore the results are not intended to be used for policy decision making. This example has been chosen as it demonstrates many of the issues in synthesising effectiveness evidence of public health interventions, including different study designs (randomised controlled trials, observational studies and cluster randomised trials), heterogeneity of populations or settings, incomplete individual participant data and complex interventions that contain multiple components.

This analysis will investigate the most effective promotional interventions for the outcome of ‘safe storage of cleaning products’ to prevent childhood poisoning accidents. There are 12 studies included in the dataset, with IPD available from nine of the studies. The covariate, single parent family, is included in the analysis to demonstrate the effect of being a single parent family on the outcome. In this example, all of the interventions are made up of one or more of the following components: education (Ed), free or low cost equipment (Eq), home safety inspection (HSI), and installation of safety equipment (In). A Bayesian approach using WinBUGS was used and therefore credible intervals (CrI) are presented with estimates of the effect sizes [ 16 ].

The original review paper by Achana et al. (2014) demonstrated pairwise meta-analysis and meta-regression using individual and cluster allocated trials, subgroup analyses, meta-regression using individual participant data (IPD) and summary aggregate data and NMA. This paper firstly applies NMA to the motivating example for context, followed by extensions to NMA.

Multiple interventions: lumping or splitting?

Often in public health there are multiple intervention options. However, interventions are often lumped together in a pairwise meta-analysis. Pairwise meta-analysis is a useful tool for two interventions or, alternatively in the presence of lumping interventions, for answering the research question: ‘are interventions in general better than a control or another group of interventions?’. However, when there are multiple interventions, this type of analysis is not appropriate for informing health care providers which intervention should be recommended to the public. ‘Lumping’ is becoming less frequent in other areas of evidence synthesis, such as for clinical interventions, as the use of sophisticated synthesis techniques, such as NMA, increases (Achana et al. 2014) but lumping is still common in public health.

NMA is an extension of the pairwise meta-analysis framework to more than two interventions. Multiple interventions that are lumped into a pairwise meta-analysis are likely to demonstrate high statistical heterogeneity. This does not mean that quantitative synthesis could not be undertaken but that a more appropriate method, NMA, should be implemented. Instead the statistical approach should be based on the research questions of the systematic review. For example, if the research question is ‘are any interventions effective for preventing obesity?’, it would be appropriate to perform a pairwise meta-analysis comparing every intervention in the literature to a control. However, if the research question is ‘which intervention is the most effective for preventing obesity?’, it would be more appropriate and informative to perform a network meta-analysis, which can compare multiple interventions simultaneously and identify the best one.

NMA is a useful statistical method in the context of public health intervention appraisal, where there are often multiple intervention options, as it estimates the relative effectiveness of three or more interventions simultaneously, even if direct study evidence is not available for all intervention comparisons. Using NMA can help to answer the research question ‘what is the effectiveness of each intervention compared to all other interventions in the network?’.

In the motivating example there are six intervention options. The effect of lumping interventions is shown in Fig.  4 , where different interventions in both the intervention and control arms are compared. There is overlap of intervention and control arms across studies and interpretation of the results of a pairwise meta-analysis comparing the effectiveness of the two groups of interventions would not be useful in deciding which intervention to recommend. In comparison, the network plot in Fig.  5 illustrates the evidence base of the prevention of childhood poisonings review comparing six interventions that promote the use of safety equipment in the home. Most of the studies use ‘usual care’ as a baseline and compare this to another intervention. There are also studies in the evidence base that compare pairs of the interventions, such as ‘Education and equipment’ to ‘Equipment’. The plot also demonstrates the absence of direct study evidence between many pairs of interventions, for which the associated treatment effects can be indirectly estimated using NMA.

Network plot to illustrate how pairwise meta-analysis groups the interventions in the motivating dataset. Notation UC: Usual care, Ed: Education, Ed+Eq: Education and equipment, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq: Equipment

Network plot for the safe storage of cleaning products outcome. Notation UC: Usual care, Ed: Education, Ed+Eq: Education and equipment, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq: Equipment

An NMA was fitted to the motivating example to compare the six interventions in the studies from the review. The results are reported in the ‘triangle table’ in Table  5 [ 17 ]. The top right half of the table shows the direct evidence between pairs of the interventions in the corresponding rows and columns by either pooling the studies as a pairwise meta-analysis or presenting the single study results if evidence is only available from a single study. The bottom left half of the table reports the results of the NMA. The gaps in the top right half of the table arise where no direct study evidence exists to compare the two interventions. For example, there is no direct study evidence comparing ‘Education’ (Ed) to ‘Education, equipment and home safety inspection’ (Ed+Eq+HSI). The NMA, however, can estimate this comparison through the direct study evidence as an odds ratio of 3.80 with a 95% credible interval of (1.16, 12.44). The results suggest that the odds of safely storing cleaning products in the Ed+Eq+HSI intervention group is 3.80 times the odds in the Ed group. The results demonstrate a key benefit of NMA that all intervention effects in a network can be estimated using indirect evidence, even if there is no direct study evidence for some pairwise comparisons. This is based on the consistency assumption (that estimates of intervention effects from direct and indirect evidence are consistent) which should be checked when performing an NMA. This is beyond the scope of this paper and details on this can be found elsewhere [ 18 ].

NMA can also be used to rank the interventions in terms of their effectiveness and estimate the probability that each intervention is likely to be the most effective. This can help to answer the research question ‘which intervention is the best?’ out of all of the interventions that have provided evidence in the network. The rankings and associated probabilities for the motivating example are presented in Table  6 . It can be seen that in this case the ‘education, equipment and home safety inspection’ (Ed+Eq+HSI) intervention is ranked first, with a 0.87 probability of being the best intervention. However, there is overlap of the 95% credible intervals of the median rankings. This overlap reflects the uncertainty in the intervention effect estimates and therefore it is important that the interpretation of these statistics clearly communicates this uncertainty to decision makers.

NMA has the potential to be extremely useful but is underutilised in the evidence synthesis of public health interventions. The ability to compare and rank multiple interventions in an area where there are often multiple intervention options is invaluable in decision making for identifying which intervention to recommend. NMA can also include further literature in the analysis, compared to a pairwise meta-analysis, by expanding the network to improve the uncertainty in the effectiveness estimates.

Statistical heterogeneity

When heterogeneity remains in the results of an NMA, it is useful to explore the reasons for this. Strategies for dealing with heterogeneity involve the inclusion of covariates in a meta-analysis or NMA to adjust for the differences in the covariates across studies [ 19 ]. Meta-regression is a statistical method developed from meta-analysis that includes covariates to potentially explain the between-study heterogeneity ‘with the aim of estimating treatment-covariate interactions’ (Saramago et al. 2012). NMA has been extended to network meta-regression which investigates the effect of trial characteristics on multiple intervention effects. Three ways have been suggested to include covariates in an NMA: single covariate effect, exchangeable covariate effects and independent covariate effects which are discussed in more detail in the NICE Technical Support Document 3 [ 14 ]. This method has the potential to assess the effect of study level covariates on the intervention effects, which is particularly relevant in public health due to the variation across studies.

The most widespread method of meta-regression uses study level data for the inclusion of covariates into meta-regression models. Study level covariate data is when the data from the studies are aggregated, e.g. the proportion of participants in a study that are from single parent families compared to dual parent families. The alternative to study level data is individual participant data (IPD), where the data are available and used as a covariate at the individual level e.g. the parental status of every individual in a study can be used as a covariate. Although IPD is considered to be the gold standard for meta-analysis, aggregated level data is much more commonly used as it is usually available and easily accessible from published research whereas IPD can be hard to obtain from study authors.

There are some limitations to network meta-regression. In our motivating example, using the single parent covariate in a meta-regression would estimate the relative difference in the intervention effects of a population that is made up of 100% single parent families compared to a population that is made up of 100% dual parent families. This interpretation is not as useful as the analysis that uses IPD, which would give the relative difference of the intervention effects in a single parent family compared to a dual parent family. The meta-regression using aggregated data would also be susceptible to ecological bias. Ecological bias is where the effect of the covariate is different at the study level compared to the individual level [ 14 ]. For example, if each study demonstrates a relationship between a covariate and the intervention but the covariate is similar across the studies, a meta-regression of the aggregate data would not demonstrate the effect that is observed within the studies [ 20 ].

Although meta-regression is a useful tool for investigating sources of heterogeneity in the data, caution should be taken when using the results of meta-regression to explain how covariates affect the intervention effects. Meta-regression should only be used to investigate study characteristics, such as the duration of intervention, which will not be susceptible to ecological bias and the interpretation of the results (the effect of intervention duration on intervention effectiveness) would be more meaningful for the development of public health interventions.

Since the covariate of interest in this motivating example is not a study characteristic, meta-regression of aggregated covariate data was not performed. Network meta-regression including IPD and aggregate level data was developed by Samarago et al. (2012) [ 21 ] to overcome the issues with aggregated data network meta-regression, which is discussed in the next section.

Tailored decision making to specific sub-groups

In public health it is important to identify which interventions are best for which people. There has been a recent move towards precision medicine. In the field of public health the ‘concept of precision prevention may [...] be valuable for efficiently targeting preventive strategies to the specific subsets of a population that will derive maximal benefit’ (Khoury and Evans, 2015). Tailoring interventions has the potential to reduce the effect of inequalities in social factors that are influencing the health of the population. Identifying which interventions should be targeted to which subgroups can also lead to better public health outcomes and help to allocate scarce NHS resources. Research interest, therefore, lies in identifying participant level covariate-intervention interactions.

IPD meta-analysis uses data at the individual level to overcome ecological bias. The interpretation of IPD meta-analysis is more relevant in the case of using participant characteristics as covariates since the interpretation of the covariate-intervention interaction is at the individual level rather than the study level. This means that it can answer the research question: ‘which interventions work best in subgroups of the population?’. IPD meta-analyses are considered to be the gold standard for evidence synthesis since it increases the power of the analysis to identify covariate-intervention interactions and it has the ability to reduce the effect of ecological bias compared to aggregated data alone. IPD meta-analysis can also help to overcome scarcity of data issues and has been shown to have higher power and reduce the uncertainty in the estimates compared to analysis including only summary aggregate data [ 22 ].

Despite the advantages of including IPD in a meta-analysis, in reality it is often very time consuming and difficult to collect IPD for all of the studies [ 21 ]. Although data sharing is becoming more common, it remains time consuming and difficult to collect IPD for all studies in a review. This results in IPD being underutilised in meta-analyses. As an intermediate solution, statistical methods have been developed, such as the NMA in Samarago et al. (2012), that incorporates both IPD and aggregate data. Methods that simultaneously include IPD and aggregate level data have been shown to reduce uncertainty in the effect estimates and minimise ecological bias [ 20 , 21 ]. A simulation study by Leahy et al. (2018) found that an increased proportion of IPD resulted in more accurate and precise NMA estimates [ 23 ].

An NMA including IPD, where it is available, was performed, based on the model presented in Samarago et al. (2012) [ 21 ]. The results in Table  7 demonstrates the detail that this type of analysis can provide to base decisions on. More relevant covariate-intervention interaction interpretations can be obtained, for example the regression coefficients for covariate-intervention interactions are the individual level covariate intervention interactions or the ‘within study interactions’ that are interpreted as the effect of being in a single parent family on the effectiveness of each of the interventions. For example, the effect of Ed+Eq compared to UC in a single parent family is 1.66 times the effect of Ed+Eq compared to UC in a dual parent family but this is not an important difference as the credible interval crosses 1. The regression coefficients for the study level covariate-intervention interactions or the ‘between study interactions’ can be interpreted as the relative difference in the intervention effects of a population that is made up of 100% single parent families compared to a population that is made up of 100% dual parent families.

• Complex interventions

In many public health research settings the complex interventions are comprised of a number of components. An NMA can compare all of the interventions in a network as they are implemented in the original trials. However, NMA does not tell us which components of the complex intervention are attributable to this effect. It could be that particular components, or the interacting effect of multiple components, are driving the effectiveness and other components are not as effective. Often, trials have not directly compared every combination of components as there are so many component combination options, it would be inefficient and impractical. Component NMA was developed by Welton et al. (2009) to estimate the effect of each component of the complex interventions and combination of components in a network, in the absence of direct trial evidence and answers the question: ‘are interventions with a particular component or combination of components effective?’ [ 11 ]. For example, for the motivating example, in comparison to Fig.  5 , which demonstrates the interventions that an NMA can estimate effectiveness, Fig.  6 demonstrates all of the possible interventions of which the effectiveness can be estimated in a component NMA, given the components present in the network.

Network plot that illustrates how component network meta-analysis can estimate the effectiveness of intervention components and combinations of components, even when they are not included in the direct evidence. Notation UC: Usual care, Ed: Education, Eq: Equipment, Installation, Ed+Eq: Education and equipment, Ed+HSI: Education and home safety inspection, Ed+In: Education and installation, Eq+HSI: Equipment and home safety inspection, Eq+In: equipment and installation, HSI+In: Home safety inspection and installation, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq+HSI+In: Equipment, home safety inspection and installation, Ed+Eq+HSI+In: Education, equipment, home safety inspection and installation

The results of the analyses of the main effects, two way effects and full effects models are shown in Table  8 . The models, proposed in the original paper by Welton et al. (2009), increase in complexity as the assumptions regarding the component effects relax [ 24 ]. The main effects component NMA assumes that the components in the interventions each have separate, independent effects and intervention effects are the sum of the component effects. The two-way effects models assumes that there are interactions between pairs of the components, so the effects of the interventions are more than the sum of the effects. The full effects model assumes that all of the components and combinations of the components interact. Component NMA did not provide further insight into which components are likely to be the most effective since all of the 95% credible intervals were very wide and overlapped 1. There is a lot of uncertainty in the results, particularly in the 2-way and full effects models. A limitation of component NMA is that there are issues with uncertainty when data is scarce. However, the results demonstrate the potential of component NMA as a useful tool to gain better insights from the available dataset.

In practice, this method has rarely been used since its development [ 24 – 26 ]. It may be challenging to define the components in some areas of public health where many interventions have been studied. However, the use of meta-analysis for planning future studies is rarely discussed and component NMA would provide a useful tool for identifying new component combinations that may be more effective [ 27 ]. This type of analysis has the potential to prioritise future public health research, which is especially useful where there are multiple intervention options, and identify more effective interventions to recommend to the public.

Further methods / other outcomes

The analysis and methods described in this paper only cover a small subset of the methods that have been developed in meta-analysis in recent years. Methods that aim to assess the quality of evidence supporting a NMA and how to quantify how much the evidence could change due to potential biases or sampling variation before the recommendation changes have been developed [ 28 , 29 ]. Models adjusting for baseline risk have been developed to allow for different study populations to have different levels of underlying risk, by using the observed event rate in the control arm [ 30 , 31 ]. Multivariate methods can be used to compare the effect of multiple interventions on two or more outcomes simultaneously [ 32 ]. This area of methodological development is especially appealing within public health where studies assess a broad range of health effects and typically have multiple outcome measures. Multivariate methods offer benefits over univariate models by allowing the borrowing of information across outcomes and modelling the relationships between outcomes which can potentially reduce the uncertainty in the effect estimates [ 33 ]. Methods have also been developed to evaluate interventions with classes or different intervention intensities, known as hierarchical interventions [ 34 ]. These methods were not demonstrated in this paper but can also be useful tools for addressing challenges of appraising public health interventions, such as multiple and surrogate outcomes.

This paper only considered an example with a binary outcome. All of the methods described have also been adapted for other outcome measures. For example, the Technical Support Document 2 proposed a Bayesian generalised linear modelling framework to synthesise other outcome measures. More information and models for continuous and time-to-event data is available elsewhere [ 21 , 35 – 38 ].

Software and guidelines

In the previous section, meta-analytic methods that answer more policy relevant questions were demonstrated. However, as shown by the update to the review, methods such as these are still under-utilised. It is suspected from the NICE public health review that one of the reasons for the lack of uptake of methods in public health could be due to common software choices, such as RevMan, being limited in their flexibility for statistical methods.

Table  9 provides a list of software options and guidance documents that are more flexible than RevMan for implementing the statistical methods illustrated in the previous section to make these methods more accessible to researchers.

In this paper, the network plot in Figs.  5 and 6 were produced using the networkplot command from the mvmeta package [ 39 ] in Stata [ 61 ]. WinBUGS was used to fit the NMA in this paper by adapting the code in the book ‘Evidence Synthesis for Decision Making in Healthcare’ which also provides more detail on Bayesian methods and assessing convergence of Bayesian models [ 45 ]. The model for including IPD and summary aggregate data in an NMA was based on the code in the paper by Saramago et al. (2012). The component NMA in this paper was performed in WinBUGS through R2WinBUGS, [ 47 ] using the code in Welton et al. (2009) [ 11 ].

WinBUGS is a flexible tool for fitting complex models in a Bayesian framework. The NICE Decision Support Unit produced a series of Evidence Synthesis Technical Support Documents [ 46 ] that provide a comprehensive technical guide to methods for evidence synthesis and WinBUGS code is also provided for many of the models. Complex models can also be performed in a frequentist framework. Code and commands for many models are available in R and STATA (see Table  9 ).

The software, R2WinBUGS, was used in the analysis of the motivating example. Increasing numbers of researchers are using R and so packages that can be used to link the two softwares by calling BUGS models in R, packages such as R2WinBUGS, can improve the accessibility of Bayesian methods [ 47 ]. The new R package, BUGSnet, may also help to facilitate the accessibility and improve the reporting of Bayesian NMA [ 48 ]. Webtools have also been developed as a means of enabling researchers to undertake increasingly complex analyses [ 52 , 53 ]. Webtools provide a user-friendly interface to perform statistical analyses and often help in the reporting of the analyses by producing plots, including network plots and forest plots. These tools are very useful for researchers that have a good understanding of the statistical methods they want to implement as part of their review but are inexperienced in statistical software.

This paper has reviewed NICE public health intervention guidelines to identify the methods that are currently being used to synthesise effectiveness evidence to inform public health decision making. A previous review from 2012 was updated to see how method utilisation has changed. Methods have been developed since the previous review and these were applied to an example dataset to show how methods can answer more policy relevant questions. Resources and guidelines for implementing these methods were signposted to encourage uptake.

The review found that the proportion of NICE guidelines containing effectiveness evidence summarised using meta-analysis methods has increased since the original review, but remains low. The majority of the reviews presented only narrative summaries of the evidence - a similar result to the original review. In recent years, there has been an increased awareness of the need to improve decision making by using all of the available evidence. As a result, this has led to the development of new methods, easier application in standard statistical software packages, and guidance documents. Based on this, it would have been expected that their implementation would rise in recent years to reflect this, but the results of the review update showed no such increasing pattern.

A high proportion of NICE guideline reports did not provide a reason for not applying quantitative evidence synthesis methods. Possible explanations for this could be time or resource constraints, lack of statistical expertise, being unaware of the available methods or poor reporting. Reporting guidelines, such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), should be updated to emphasise the importance of documenting reasons for not applying methods, as this can direct future research to improve uptake.

Where it was specified, the most common reported reason for not conducting a meta-analysis was heterogeneity. Often in public health, the data is heterogeneous due to the differences between studies in population, design, interventions or outcomes. A common misconception is that the presence of heterogeneity implies that it is not possible to pool the data. Meta-analytic methods can be used to investigate the sources of heterogeneity, as demonstrated in the NMA of the motivating example, and the use of IPD is recommended where possible to improve the precision of the results and reduce the effect of ecological bias. Although caution should be exercised in the interpretation of the results, quantitative synthesis methods provide a stronger basis for making decisions than narrative accounts because they explicitly quantify the heterogeneity and seek to explain it where possible.

The review also found that the most common software to perform the synthesis was RevMan. RevMan is very limited in its ability to perform advanced statistical analyses, beyond that of pairwise meta-analysis, which might explain the above findings. Standard software code is being developed to help make statistical methodology and application more accessible and guidance documents are becoming increasingly available.

The evaluation of public health interventions can be problematic due to the number and complexity of the interventions. NMA methods were applied to a real Cochrane public health review dataset. The methods that were demonstrated showed ways to address some of these issues, including the use of NMA for multiple interventions, the inclusion of covariates as both aggregated data and IPD to explain heterogeneity, and the extension to component network meta-analysis for guiding future research. These analyses illustrated how the choice of synthesis methods can enable more informed decision making by allowing more distinct interventions, and combinations of intervention components, to be defined and their effectiveness estimated. It also demonstrated the potential to target interventions to population subgroups where they are likely to be most effective. However, the application of component NMA to the motivating example has also demonstrated the issues around uncertainty if there are a limited number of studies observing the interventions and intervention components.

The application of methods to the motivating example demonstrated a key benefit of using statistical methods in a public health context compared to only presenting a narrative review – the methods provide a quantitative estimate of the effectiveness of the interventions. The uncertainty from the credible intervals can be used to demonstrate the lack of available evidence. In the context of decision making, having pooled estimates makes it much easier for decision makers to assess the effectiveness of the interventions or identify when more research is required. The posterior distribution of the pooled results from the evidence synthesis can also be incorporated into a comprehensive decision analytic model to determine cost-effectiveness [ 62 ]. Although narrative reviews are useful for describing the evidence base, the results are very difficult to summarise in a decision context.

Although heterogeneity seems to be inevitable within public health interventions due to their complex nature, this review has shown that it is still the main reported reason for not using statistical methods in evidence synthesis. This may be due to guidelines that were originally developed for clinical treatments that are tested in randomised conditions still being applied in public health settings. Guidelines for the choice of methods used in public health intervention appraisals could be updated to take into account the complexities and wide ranging areas in public health. Sophisticated methods may be more appropriate in some cases than simpler models for modelling multiple, complex interventions and their uncertainty, given the limitations are also fully reported [ 19 ]. Synthesis may not be appropriate if statistical heterogeneity remains after adjustment for possible explanatory covariates but details of exploratory analysis and reasons for not synthesising the data should be reported. Future research should focus on the application and dissemination of the advantages of using more advanced methods in public health, identifying circumstances where these methods are likely to be the most beneficial, and ways to make the methods more accessible, for example, the development of packages and web tools.

There is an evident need to facilitate the translation of the synthesis methods into a public health context and encourage the use of methods to improve decision making. This review has shown that the uptake of statistical methods for evaluating the effectiveness of public health interventions is slow, despite advances in methods that address specific issues in public health intervention appraisal and the publication of guidance documents to complement their application.

Availability of data and materials

The dataset supporting the conclusions of this article is included within the article.

Abbreviations

National institute for health and care excellence

• Network meta-analysis

Individual participant data

Home safety inspection

Installation

Credible interval

Preferred reporting items for systematic reviews and meta-analyses

Dias S, Welton NJ, Sutton AJ, Ades A. NICE DSU Technical Support Document 2: A Generalised Linear Modelling Framework for Pairwise and Network Meta-Analysis of Randomised Controlled Trials: National Institute for Health and Clinical Excellence; 2011, p. 98. (Technical Support Document in Evidence Synthesis; TSD2).

Higgins JPT, López-López JA, Becker BJ, et al.Synthesising quantitative evidence in systematic reviews of complex health interventions. BMJ Global Health. 2019; 4(Suppl 1):e000858. https://doi.org/10.1136/bmjgh-2018-000858 .

Article   PubMed   PubMed Central   Google Scholar  

Achana F, Hubbard S, Sutton A, Kendrick D, Cooper N. An exploration of synthesis methods in public health evaluations of interventions concludes that the use of modern statistical methods would be beneficial. J Clin Epidemiol. 2014; 67(4):376–90.

Article   PubMed   Google Scholar  

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new medical research council guidance. Int J Nurs Stud. 2013; 50(5):587–92.

Caldwell DM, Welton NJ. Approaches for synthesising complex mental health interventions in meta-analysis. Evidence-Based Mental Health. 2016; 19(1):16–21.

Melendez-Torres G, Bonell C, Thomas J. Emergent approaches to the meta-analysis of multiple heterogeneous complex interventions. BMC Med Res Methodol. 2015; 15(1):47.

Article   CAS   PubMed   PubMed Central   Google Scholar  

NICE. NICE: Who We Are. https://www.nice.org.uk/about/who-we-are . Accessed 19 Sept 2019.

Kelly M, Morgan A, Ellis S, Younger T, Huntley J, Swann C. Evidence based public health: a review of the experience of the national institute of health and clinical excellence (NICE) of developing public health guidance in England. Soc Sci Med. 2010; 71(6):1056–62.

NICE. Developing NICE Guidelines: The Manual. https://www.nice.org.uk/process/pmg20/chapter/introduction-and-overview . Accessed 19 Sept 2019.

NICE. Public Health Guidance. https://www.nice.org.uk/guidance/published?type=ph . Accessed 19 Sept 2019.

Welton NJ, Caldwell D, Adamopoulos E, Vedhara K. Mixed treatment comparison meta-analysis of complex interventions: psychological interventions in coronary heart disease. Am J Epidemiol. 2009; 169(9):1158–65.

Ioannidis JP, Patsopoulos NA, Rothstein HR. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ. 2008; 336(7658):1413–5.

Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002; 21(11):1539–58.

Article   Google Scholar  

Dias S, Sutton A, Welton N, Ades A. NICE DSU Technical Support Document 3: Heterogeneity: Subgroups, Meta-Regression, Bias and Bias-Adjustment: National Institute for Health and Clinical Excellence; 2011, p. 76.

Kendrick D, Ablewhite J, Achana F, et al.Keeping Children Safe: a multicentre programme of research to increase the evidence base for preventing unintentional injuries in the home in the under-fives. Southampton: NIHR Journals Library; 2017.

Google Scholar  

Lunn DJ, Thomas A, Best N, et al.WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility. Stat Comput. 2000; 10:325–37. https://doi.org/10.1023/A:1008929526011 .

Dias S, Caldwell DM. Network meta-analysis explained. Arch Dis Child Fetal Neonatal Ed. 2019; 104(1):8–12. https://doi.org/10.1136/archdischild-2018-315224. http://arxiv.org/abs/https://fn.bmj.com/content/104/1/F8.full.pdf.

Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades A. NICE DSU Technical Support Document 4: Inconsistency in Networks of Evidence Based on Randomised Controlled Trials: National Institute for Health and Clinical Excellence; 2011. (NICE DSU Technical Support Document in Evidence Synthesis; TSD4).

Cipriani A, Higgins JP, Geddes JR, Salanti G. Conceptual and technical challenges in network meta-analysis. Ann Intern Med. 2013; 159(2):130–7.

Riley RD, Steyerberg EW. Meta-analysis of a binary outcome using individual participant data and aggregate data. Res Synth Methods. 2010; 1(1):2–19.

Saramago P, Sutton AJ, Cooper NJ, Manca A. Mixed treatment comparisons using aggregate and individual participant level data. Stat Med. 2012; 31(28):3516–36.

Lambert PC, Sutton AJ, Abrams KR, Jones DR. A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol. 2002; 55(1):86–94.

Article   CAS   PubMed   Google Scholar  

Leahy J, O’Leary A, Afdhal N, Gray E, Milligan S, Wehmeyer MH, Walsh C. The impact of individual patient data in a network meta-analysis: an investigation into parameter estimation and model selection. Res Synth Methods. 2018; 9(3):441–69.

Freeman SC, Scott NW, Powell R, Johnston M, Sutton AJ, Cooper NJ. Component network meta-analysis identifies the most effective components of psychological preparation for adults undergoing surgery under general anesthesia. J Clin Epidemiol. 2018; 98:105–16.

Pompoli A, Furukawa TA, Efthimiou O, Imai H, Tajika A, Salanti G. Dismantling cognitive-behaviour therapy for panic disorder: a systematic review and component network meta-analysis. Psychol Med. 2018; 48(12):1945–53.

Rücker G, Schmitz S, Schwarzer G. Component network meta-analysis compared to a matching method in a disconnected network: A case study. Biom J. 2020. https://doi.org/10.1002/bimj.201900339 .

Efthimiou O, Debray TP, van Valkenhoef G, Trelle S, Panayidou K, Moons KG, Reitsma JB, Shang A, Salanti G, Group GMR. GetReal in network meta-analysis: a review of the methodology. Res Synth Methods. 2016; 7(3):236–63.

Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network meta-analysis. PLoS ONE. 2014; 9(7):99682.

Article   CAS   Google Scholar  

Phillippo DM, Dias S, Welton NJ, Caldwell DM, Taske N, Ades A. Threshold analysis as an alternative to grade for assessing confidence in guideline recommendations based on network meta-analyses. Ann Intern Med. 2019; 170(8):538–46.

Dias S, Welton NJ, Sutton AJ, Ades AE. NICE DSU Technical Support Document 5: Evidence Synthesis in the Baseline Natural History Model: National Institute for Health and Clinical Excellence; 2011, p. 29. (NICE DSU Technical Support Document in Evidence Synthesis; TSD5).

Achana FA, Cooper NJ, Dias S, Lu G, Rice SJ, Kendrick D, Sutton AJ. Extending methods for investigating the relationship between treatment effect and baseline risk from pairwise meta-analysis to network meta-analysis. Stat Med. 2013; 32(5):752–71.

Riley RD, Jackson D, Salanti G, Burke DL, Price M, Kirkham J, White IR. Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples. BMJ (Clinical research ed.) 2017; 358:j3932. https://doi.org/10.1136/bmj.j3932 .

Achana FA, Cooper NJ, Bujkiewicz S, Hubbard SJ, Kendrick D, Jones DR, Sutton AJ. Network meta-analysis of multiple outcome measures accounting for borrowing of information across outcomes. BMC Med Res Methodol. 2014; 14(1):92.

Owen RK, Tincello DG, Keith RA. Network meta-analysis: development of a three-level hierarchical modeling approach incorporating dose-related constraints. Value Health. 2015; 18(1):116–26.

Jansen JP. Network meta-analysis of individual and aggregate level data. Res Synth Methods. 2012; 3(2):177–90.

Donegan S, Williamson P, D’Alessandro U, Garner P, Smith CT. Combining individual patient data and aggregate data in mixed treatment comparison meta-analysis: individual patient data may be beneficial if only for a subset of trials. Stat Med. 2013; 32(6):914–30.

Saramago P, Chuang L-H, Soares MO. Network meta-analysis of (individual patient) time to event data alongside (aggregate) count data. BMC Med Res Methodol. 2014; 14(1):105.

Thom HH, Capkun G, Cerulli A, Nixon RM, Howard LS. Network meta-analysis combining individual patient and aggregate data from a mixture of study designs with an application to pulmonary arterial hypertension. BMC Med Res Methodol. 2015; 15(1):34.

Gasparrini A, Armstrong B, Kenward MG. Multivariate meta-analysis for non-linear and other multi-parameter associations. Stat Med. 2012; 31(29):3821–39.

Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graphical tools for network meta-analysis in stata. PLoS ONE. 2013; 8(10):76654.

Rücker G, Schwarzer G, Krahn U, König J. netmeta: Network meta-analysis with R. R package version 0.5-0. 2014. R package version 0.5-0. Availiable: http://CRAN.R-project.org/package=netmeta .

van Valkenhoef G, Kuiper J. gemtc: Network Meta-Analysis Using Bayesian Methods. R package version 0.8-2. 2016. Available online at: https://CRAN.R-project.org/package=gemtc .

Lin L, Zhang J, Hodges JS, Chu H. Performing arm-based network meta-analysis in R with the pcnetmeta package. J Stat Softw. 2017; 80(5):1–25. https://doi.org/10.18637/jss.v080.i05 .

Rücker G, Schwarzer G. Automated drawing of network plots in network meta-analysis. Res Synth Methods. 2016; 7(1):94–107.

Welton NJ, Sutton AJ, Cooper N, Abrams KR, Ades A. Evidence Synthesis for Decision Making in Healthcare, vol. 132. UK: Wiley; 2012.

Book   Google Scholar  

Dias S, Welton NJ, Sutton AJ, Ades AE. Evidence synthesis for decision making 1: introduction. Med Decis Making Int J Soc Med Decis Making. 2013; 33(5):597–606. https://doi.org/10.1177/0272989X13487604 .

Sturtz S, Ligges U, Gelman A. R2WinBUGS: a package for running WinBUGS from R. J Stat Softw. 2005; 12(3):1–16.

Béliveau A, Boyne DJ, Slater J, Brenner D, Arora P. Bugsnet: an r package to facilitate the conduct and reporting of bayesian network meta-analyses. BMC Med Res Methodol. 2019; 19(1):196.

Neupane B, Richer D, Bonner AJ, Kibret T, Beyene J. Network meta-analysis using R: a review of currently available automated packages. PLoS ONE. 2014; 9(12):115065.

White IR. Multivariate random-effects meta-analysis. Stata J. 2009; 9(1):40–56.

Chaimani A, Salanti G. Visualizing assumptions and results in network meta-analysis: the network graphs package. Stata J. 2015; 15(4):905–50.

Owen RK, Bradbury N, Xin Y, Cooper N, Sutton A. MetaInsight: An interactive web-based tool for analyzing, interrogating, and visualizing network meta-analyses using R-shiny and netmeta. Res Synth Methods. 2019; 10(4):569–81. https://doi.org/10.1002/jrsm.1373 .

Freeman SC, Kerby CR, Patel A, Cooper NJ, Quinn T, Sutton AJ. Development of an interactive web-based tool to conduct and interrogate meta-analysis of diagnostic test accuracy studies: MetaDTA. BMC Med Res Methodol. 2019; 19(1):81.

Nikolakopoulou A, Higgins JPT, Papakonstantinou T, Chaimani A, Del Giovane C, Egger M, Salanti G. CINeMA: An approach for assessing confidence in the results of a network meta-analysis. PLoS Med. 2020; 17(4):e1003082. https://doi.org/10.1371/journal.pmed.1003082 .

Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010; 36(3):1–48.

Freeman SC, Carpenter JR. Bayesian one-step ipd network meta-analysis of time-to-event data using royston-parmar models. Res Synth Methods. 2017; 8(4):451–64.

Riley RD, Lambert PC, Staessen JA, Wang J, Gueyffier F, Thijs L, Boutitie F. Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Stat Med. 2008; 27(11):1870–93.

Debray TP, Moons KG, van Valkenhoef G, Efthimiou O, Hummel N, Groenwold RH, Reitsma JB, Group GMR. Get real in individual participant data (ipd) meta-analysis: a review of the methodology. Res Synth Methods. 2015; 6(4):293–309.

Tierney JF, Vale C, Riley R, Smith CT, Stewart L, Clarke M, Rovers M. Individual Participant Data (IPD) Meta-analyses of Randomised Controlled Trials: Guidance on Their Use. PLoS Med. 2015; 12(7):e1001855. https://doi.org/10.1371/journal.pmed.1001855 .

Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, Tierney JF. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the prisma-ipd statement. JAMA. 2015; 313(16):1657–65.

StataCorp. Stata Statistical Software: Release 16. College Station: StataCorp LLC; 2019.

Cooper NJ, Sutton AJ, Abrams KR, Turner D, Wailoo A. Comprehensive decision analytical modelling in economic evaluation: a bayesian approach. Health Econ. 2004; 13(3):203–26.

Download references

Acknowledgements

We would like to acknowledge Professor Denise Kendrick as the lead on the NIHR Keeping Children Safe at Home Programme that originally funded the collection of the evidence for the motivating example and some of the analyses illustrated in the paper.

ES is funded by a National Institute for Health Research (NIHR), Doctoral Research Fellow for this research project. This paper presents independent research funded by the National Institute for Health Research (NIHR). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Author information

Authors and affiliations.

Department of Health Sciences, University of Leicester, Lancaster Road, Leicester, UK

Ellesha A. Smith, Nicola J. Cooper, Alex J. Sutton, Keith R. Abrams & Stephanie J. Hubbard

You can also search for this author in PubMed   Google Scholar

Contributions

ES performed the review, analysed the data and wrote the paper. SH supervised the project. SH, KA, NC and AS provided substantial feedback on the manuscript. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Ellesha A. Smith .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

KA is supported by Health Data Research (HDR) UK, the UK National Institute for Health Research (NIHR) Applied Research Collaboration East Midlands (ARC EM), and as a NIHR Senior Investigator Emeritus (NF-SI-0512-10159). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. KA has served as a paid consultant, providing unrelated methodological advice, to; Abbvie, Amaris, Allergan, Astellas, AstraZeneca, Boehringer Ingelheim, Bristol-Meyers Squibb, Creativ-Ceutical, GSK, ICON/Oxford Outcomes, Ipsen, Janssen, Eli Lilly, Merck, NICE, Novartis, NovoNordisk, Pfizer, PRMA, Roche and Takeda, and has received research funding from Association of the British Pharmaceutical Industry (ABPI), European Federation of Pharmaceutical Industries & Associations (EFPIA), Pfizer, Sanofi and Swiss Precision Diagnostics. He is a Partner and Director of Visible Analytics Limited, a healthcare consultancy company.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Key for the Nice public health guideline codes. Available in NICEGuidelinesKey.xlsx .

Additional file 2

NICE public health intervention guideline review flowchart for the inclusion and exclusion of documents. Available in Flowchart.JPG .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Smith, E.A., Cooper, N.J., Sutton, A.J. et al. A review of the quantitative effectiveness evidence synthesis methods used in public health intervention guidelines. BMC Public Health 21 , 278 (2021). https://doi.org/10.1186/s12889-021-10162-8

Download citation

Received : 22 September 2020

Accepted : 04 January 2021

Published : 03 February 2021

DOI : https://doi.org/10.1186/s12889-021-10162-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Meta-analysis
• Systematic review
• Public health
• Decision making
• Evidence synthesis

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

PH717 Module 1B - Descriptive Tools

Descriptive epidemiology & descriptive statistics.

•   Page:
•   1  
• |   2  
• |   3  
• |   4  
• |   5  
• |   6  
• |   7  
• |   8  
• |   9  

Quantitative and Qualitative Methods for Public Health

This course focuses on quantitative methods, which are designed to precisely estimate population parameters and measure the association between biologic, social, environmental, and behavioral factors and health conditions in order to define the determinants of health and disease and, ultimately, to understand causal pathways.

However, it is important to acknowledge the importance of qualitative methods which provide a means of understanding public health problems in greater depth by providing contextual information regarding a population's beliefs, opinions, norms, and behaviors. This type of information is difficult to capture using traditional quantitative methods, yet it can be vitally important for understanding the "why" for many health problems and also the "how" in terms of how to achieve improvements in health outcomes.

These two approaches might be thought of as the positivist and the constructivist approaches. In positivist research data are more easily quantified, but they are disconnected from the context in which they occur. For example, people of lower socioeconomic status are more likely to smoke tobacco, but the data collected does not indicate why. However, with a constructivist approach, the exposures that people are subjected to (or choose) are better understood in the context of their personal circumstances and the significance that people attribute to things in their environment.

Qualitative methods provide a means of understanding health problems and potential barriers and solutions in greater detail, and they provide an opportunity to understand the "how" and "why" and to identify overlooked issues and themes.

The table below, from the introductory course on fundamentals of public health, highlights some of the major differences between quantitative and qualitative research.

return to top | previous page

Content ©2020. All Rights Reserved. Date last modified: September 10, 2020. Wayne W. LaMorte, MD, PhD, MPH

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Recent quantitative research on determinants of health in high income countries: A scoping review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Centre for Health Economics Research and Modelling Infectious Diseases, Vaccine and Infectious Disease Institute, University of Antwerp, Antwerp, Belgium

Roles Conceptualization, Data curation, Funding acquisition, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

• Vladimira Varbanova, 
• Philippe Beutels

• Published: September 17, 2020
• https://doi.org/10.1371/journal.pone.0239031
• Peer Review
• Reader Comments

Identifying determinants of health and understanding their role in health production constitutes an important research theme. We aimed to document the state of recent multi-country research on this theme in the literature.

We followed the PRISMA-ScR guidelines to systematically identify, triage and review literature (January 2013—July 2019). We searched for studies that performed cross-national statistical analyses aiming to evaluate the impact of one or more aggregate level determinants on one or more general population health outcomes in high-income countries. To assess in which combinations and to what extent individual (or thematically linked) determinants had been studied together, we performed multidimensional scaling and cluster analysis.

Sixty studies were selected, out of an original yield of 3686. Life-expectancy and overall mortality were the most widely used population health indicators, while determinants came from the areas of healthcare, culture, politics, socio-economics, environment, labor, fertility, demographics, life-style, and psychology. The family of regression models was the predominant statistical approach. Results from our multidimensional scaling showed that a relatively tight core of determinants have received much attention, as main covariates of interest or controls, whereas the majority of other determinants were studied in very limited contexts. We consider findings from these studies regarding the importance of any given health determinant inconclusive at present. Across a multitude of model specifications, different country samples, and varying time periods, effects fluctuated between statistically significant and not significant, and between beneficial and detrimental to health.

Conclusions

We conclude that efforts to understand the underlying mechanisms of population health are far from settled, and the present state of research on the topic leaves much to be desired. It is essential that future research considers multiple factors simultaneously and takes advantage of more sophisticated methodology with regards to quantifying health as well as analyzing determinants’ influence.

Citation: Varbanova V, Beutels P (2020) Recent quantitative research on determinants of health in high income countries: A scoping review. PLoS ONE 15(9): e0239031. https://doi.org/10.1371/journal.pone.0239031

Editor: Amir Radfar, University of Central Florida, UNITED STATES

Received: November 14, 2019; Accepted: August 28, 2020; Published: September 17, 2020

Copyright: © 2020 Varbanova, Beutels. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: This study (and VV) is funded by the Research Foundation Flanders ( https://www.fwo.be/en/ ), FWO project number G0D5917N, award obtained by PB. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Identifying the key drivers of population health is a core subject in public health and health economics research. Between-country comparative research on the topic is challenging. In order to be relevant for policy, it requires disentangling different interrelated drivers of “good health”, each having different degrees of importance in different contexts.

“Good health”–physical and psychological, subjective and objective–can be defined and measured using a variety of approaches, depending on which aspect of health is the focus. A major distinction can be made between health measurements at the individual level or some aggregate level, such as a neighborhood, a region or a country. In view of this, a great diversity of specific research topics exists on the drivers of what constitutes individual or aggregate “good health”, including those focusing on health inequalities, the gender gap in longevity, and regional mortality and longevity differences.

The current scoping review focuses on determinants of population health. Stated as such, this topic is quite broad. Indeed, we are interested in the very general question of what methods have been used to make the most of increasingly available region or country-specific databases to understand the drivers of population health through inter-country comparisons. Existing reviews indicate that researchers thus far tend to adopt a narrower focus. Usually, attention is given to only one health outcome at a time, with further geographical and/or population [ 1 , 2 ] restrictions. In some cases, the impact of one or more interventions is at the core of the review [ 3 – 7 ], while in others it is the relationship between health and just one particular predictor, e.g., income inequality, access to healthcare, government mechanisms [ 8 – 13 ]. Some relatively recent reviews on the subject of social determinants of health [ 4 – 6 , 14 – 17 ] have considered a number of indicators potentially influencing health as opposed to a single one. One review defines “social determinants” as “the social, economic, and political conditions that influence the health of individuals and populations” [ 17 ] while another refers even more broadly to “the factors apart from medical care” [ 15 ].

In the present work, we aimed to be more inclusive, setting no limitations on the nature of possible health correlates, as well as making use of a multitude of commonly accepted measures of general population health. The goal of this scoping review was to document the state of the art in the recent published literature on determinants of population health, with a particular focus on the types of determinants selected and the methodology used. In doing so, we also report the main characteristics of the results these studies found. The materials collected in this review are intended to inform our (and potentially other researchers’) future analyses on this topic. Since the production of health is subject to the law of diminishing marginal returns, we focused our review on those studies that included countries where a high standard of wealth has been achieved for some time, i.e., high-income countries belonging to the Organisation for Economic Co-operation and Development (OECD) or Europe. Adding similar reviews for other country income groups is of limited interest to the research we plan to do in this area.

In view of its focus on data and methods, rather than results, a formal protocol was not registered prior to undertaking this review, but the procedure followed the guidelines of the PRISMA statement for scoping reviews [ 18 ].

We focused on multi-country studies investigating the potential associations between any aggregate level (region/city/country) determinant and general measures of population health (e.g., life expectancy, mortality rate).

Within the query itself, we listed well-established population health indicators as well as the six world regions, as defined by the World Health Organization (WHO). We searched only in the publications’ titles in order to keep the number of hits manageable, and the ratio of broadly relevant abstracts over all abstracts in the order of magnitude of 10% (based on a series of time-focused trial runs). The search strategy was developed iteratively between the two authors and is presented in S1 Appendix . The search was performed by VV in PubMed and Web of Science on the 16 th of July, 2019, without any language restrictions, and with a start date set to the 1 st of January, 2013, as we were interested in the latest developments in this area of research.

Eligibility criteria

Records obtained via the search methods described above were screened independently by the two authors. Consistency between inclusion/exclusion decisions was approximately 90% and the 43 instances where uncertainty existed were judged through discussion. Articles were included subject to meeting the following requirements: (a) the paper was a full published report of an original empirical study investigating the impact of at least one aggregate level (city/region/country) factor on at least one health indicator (or self-reported health) of the general population (the only admissible “sub-populations” were those based on gender and/or age); (b) the study employed statistical techniques (calculating correlations, at the very least) and was not purely descriptive or theoretical in nature; (c) the analysis involved at least two countries or at least two regions or cities (or another aggregate level) in at least two different countries; (d) the health outcome was not differentiated according to some socio-economic factor and thus studied in terms of inequality (with the exception of gender and age differentiations); (e) mortality, in case it was one of the health indicators under investigation, was strictly “total” or “all-cause” (no cause-specific or determinant-attributable mortality).

Data extraction

The following pieces of information were extracted in an Excel table from the full text of each eligible study (primarily by VV, consulting with PB in case of doubt): health outcome(s), determinants, statistical methodology, level of analysis, results, type of data, data sources, time period, countries. The evidence is synthesized according to these extracted data (often directly reflected in the section headings), using a narrative form accompanied by a “summary-of-findings” table and a graph.

Search and selection

The initial yield contained 4583 records, reduced to 3686 after removal of duplicates ( Fig 1 ). Based on title and abstract screening, 3271 records were excluded because they focused on specific medical condition(s) or specific populations (based on morbidity or some other factor), dealt with intervention effectiveness, with theoretical or non-health related issues, or with animals or plants. Of the remaining 415 papers, roughly half were disqualified upon full-text consideration, mostly due to using an outcome not of interest to us (e.g., health inequality), measuring and analyzing determinants and outcomes exclusively at the individual level, performing analyses one country at a time, employing indices that are a mixture of both health indicators and health determinants, or not utilizing potential health determinants at all. After this second stage of the screening process, 202 papers were deemed eligible for inclusion. This group was further dichotomized according to level of economic development of the countries or regions under study, using membership of the OECD or Europe as a reference “cut-off” point. Sixty papers were judged to include high-income countries, and the remaining 142 included either low- or middle-income countries or a mix of both these levels of development. The rest of this report outlines findings in relation to high-income countries only, reflecting our own primary research interests. Nonetheless, we chose to report our search yield for the other income groups for two reasons. First, to gauge the relative interest in applied published research for these different income levels; and second, to enable other researchers with a focus on determinants of health in other countries to use the extraction we made here.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0239031.g001

Health outcomes

The most frequent population health indicator, life expectancy (LE), was present in 24 of the 60 studies. Apart from “life expectancy at birth” (representing the average life-span a newborn is expected to have if current mortality rates remain constant), also called “period LE” by some [ 19 , 20 ], we encountered as well LE at 40 years of age [ 21 ], at 60 [ 22 ], and at 65 [ 21 , 23 , 24 ]. In two papers, the age-specificity of life expectancy (be it at birth or another age) was not stated [ 25 , 26 ].

Some studies considered male and female LE separately [ 21 , 24 , 25 , 27 – 33 ]. This consideration was also often observed with the second most commonly used health index [ 28 – 30 , 34 – 38 ]–termed “total”, or “overall”, or “all-cause”, mortality rate (MR)–included in 22 of the 60 studies. In addition to gender, this index was also sometimes broken down according to age group [ 30 , 39 , 40 ], as well as gender-age group [ 38 ].

While the majority of studies under review here focused on a single health indicator, 23 out of the 60 studies made use of multiple outcomes, although these outcomes were always considered one at a time, and sometimes not all of them fell within the scope of our review. An easily discernable group of indices that typically went together [ 25 , 37 , 41 ] was that of neonatal (deaths occurring within 28 days postpartum), perinatal (fetal or early neonatal / first-7-days deaths), and post-neonatal (deaths between the 29 th day and completion of one year of life) mortality. More often than not, these indices were also accompanied by “stand-alone” indicators, such as infant mortality (deaths within the first year of life; our third most common index found in 16 of the 60 studies), maternal mortality (deaths during pregnancy or within 42 days of termination of pregnancy), and child mortality rates. Child mortality has conventionally been defined as mortality within the first 5 years of life, thus often also called “under-5 mortality”. Nonetheless, Pritchard & Wallace used the term “child mortality” to denote deaths of children younger than 14 years [ 42 ].

As previously stated, inclusion criteria did allow for self-reported health status to be used as a general measure of population health. Within our final selection of studies, seven utilized some form of subjective health as an outcome variable [ 25 , 43 – 48 ]. Additionally, the Health Human Development Index [ 49 ], healthy life expectancy [ 50 ], old-age survival [ 51 ], potential years of life lost [ 52 ], and disability-adjusted life expectancy [ 25 ] were also used.

We note that while in most cases the indicators mentioned above (and/or the covariates considered, see below) were taken in their absolute or logarithmic form, as a—typically annual—number, sometimes they were used in the form of differences, change rates, averages over a given time period, or even z-scores of rankings [ 19 , 22 , 40 , 42 , 44 , 53 – 57 ].

Regions, countries, and populations

Despite our decision to confine this review to high-income countries, some variation in the countries and regions studied was still present. Selection seemed to be most often conditioned on the European Union, or the European continent more generally, and the Organisation of Economic Co-operation and Development (OECD), though, typically, not all member nations–based on the instances where these were also explicitly listed—were included in a given study. Some of the stated reasons for omitting certain nations included data unavailability [ 30 , 45 , 54 ] or inconsistency [ 20 , 58 ], Gross Domestic Product (GDP) too low [ 40 ], differences in economic development and political stability with the rest of the sampled countries [ 59 ], and national population too small [ 24 , 40 ]. On the other hand, the rationales for selecting a group of countries included having similar above-average infant mortality [ 60 ], similar healthcare systems [ 23 ], and being randomly drawn from a social spending category [ 61 ]. Some researchers were interested explicitly in a specific geographical region, such as Eastern Europe [ 50 ], Central and Eastern Europe [ 48 , 60 ], the Visegrad (V4) group [ 62 ], or the Asia/Pacific area [ 32 ]. In certain instances, national regions or cities, rather than countries, constituted the units of investigation instead [ 31 , 51 , 56 , 62 – 66 ]. In two particular cases, a mix of countries and cities was used [ 35 , 57 ]. In another two [ 28 , 29 ], due to the long time periods under study, some of the included countries no longer exist. Finally, besides “European” and “OECD”, the terms “developed”, “Western”, and “industrialized” were also used to describe the group of selected nations [ 30 , 42 , 52 , 53 , 67 ].

As stated above, it was the health status of the general population that we were interested in, and during screening we made a concerted effort to exclude research using data based on a more narrowly defined group of individuals. All studies included in this review adhere to this general rule, albeit with two caveats. First, as cities (even neighborhoods) were the unit of analysis in three of the studies that made the selection [ 56 , 64 , 65 ], the populations under investigation there can be more accurately described as general urban , instead of just general. Second, oftentimes health indicators were stratified based on gender and/or age, therefore we also admitted one study that, due to its specific research question, focused on men and women of early retirement age [ 35 ] and another that considered adult males only [ 68 ].

Data types and sources

A great diversity of sources was utilized for data collection purposes. The accessible reference databases of the OECD ( https://www.oecd.org/ ), WHO ( https://www.who.int/ ), World Bank ( https://www.worldbank.org/ ), United Nations ( https://www.un.org/en/ ), and Eurostat ( https://ec.europa.eu/eurostat ) were among the top choices. The other international databases included Human Mortality [ 30 , 39 , 50 ], Transparency International [ 40 , 48 , 50 ], Quality of Government [ 28 , 69 ], World Income Inequality [ 30 ], International Labor Organization [ 41 ], International Monetary Fund [ 70 ]. A number of national databases were referred to as well, for example the US Bureau of Statistics [ 42 , 53 ], Korean Statistical Information Services [ 67 ], Statistics Canada [ 67 ], Australian Bureau of Statistics [ 67 ], and Health New Zealand Tobacco control and Health New Zealand Food and Nutrition [ 19 ]. Well-known surveys, such as the World Values Survey [ 25 , 55 ], the European Social Survey [ 25 , 39 , 44 ], the Eurobarometer [ 46 , 56 ], the European Value Survey [ 25 ], and the European Statistics of Income and Living Condition Survey [ 43 , 47 , 70 ] were used as data sources, too. Finally, in some cases [ 25 , 28 , 29 , 35 , 36 , 41 , 69 ], built-for-purpose datasets from previous studies were re-used.

In most of the studies, the level of the data (and analysis) was national. The exceptions were six papers that dealt with Nomenclature of Territorial Units of Statistics (NUTS2) regions [ 31 , 62 , 63 , 66 ], otherwise defined areas [ 51 ] or cities [ 56 ], and seven others that were multilevel designs and utilized both country- and region-level data [ 57 ], individual- and city- or country-level [ 35 ], individual- and country-level [ 44 , 45 , 48 ], individual- and neighborhood-level [ 64 ], and city-region- (NUTS3) and country-level data [ 65 ]. Parallel to that, the data type was predominantly longitudinal, with only a few studies using purely cross-sectional data [ 25 , 33 , 43 , 45 – 48 , 50 , 62 , 67 , 68 , 71 , 72 ], albeit in four of those [ 43 , 48 , 68 , 72 ] two separate points in time were taken (thus resulting in a kind of “double cross-section”), while in another the averages across survey waves were used [ 56 ].

In studies using longitudinal data, the length of the covered time periods varied greatly. Although this was almost always less than 40 years, in one study it covered the entire 20 th century [ 29 ]. Longitudinal data, typically in the form of annual records, was sometimes transformed before usage. For example, some researchers considered data points at 5- [ 34 , 36 , 49 ] or 10-year [ 27 , 29 , 35 ] intervals instead of the traditional 1, or took averages over 3-year periods [ 42 , 53 , 73 ]. In one study concerned with the effect of the Great Recession all data were in a “recession minus expansion change in trends”-form [ 57 ]. Furthermore, there were a few instances where two different time periods were compared to each other [ 42 , 53 ] or when data was divided into 2 to 4 (possibly overlapping) periods which were then analyzed separately [ 24 , 26 , 28 , 29 , 31 , 65 ]. Lastly, owing to data availability issues, discrepancies between the time points or periods of data on the different variables were occasionally observed [ 22 , 35 , 42 , 53 – 55 , 63 ].

Health determinants

Together with other essential details, Table 1 lists the health correlates considered in the selected studies. Several general categories for these correlates can be discerned, including health care, political stability, socio-economics, demographics, psychology, environment, fertility, life-style, culture, labor. All of these, directly or implicitly, have been recognized as holding importance for population health by existing theoretical models of (social) determinants of health [ 74 – 77 ].

https://doi.org/10.1371/journal.pone.0239031.t001

It is worth noting that in a few studies there was just a single aggregate-level covariate investigated in relation to a health outcome of interest to us. In one instance, this was life satisfaction [ 44 ], in another–welfare system typology [ 45 ], but also gender inequality [ 33 ], austerity level [ 70 , 78 ], and deprivation [ 51 ]. Most often though, attention went exclusively to GDP [ 27 , 29 , 46 , 57 , 65 , 71 ]. It was often the case that research had a more particular focus. Among others, minimum wages [ 79 ], hospital payment schemes [ 23 ], cigarette prices [ 63 ], social expenditure [ 20 ], residents’ dissatisfaction [ 56 ], income inequality [ 30 , 69 ], and work leave [ 41 , 58 ] took center stage. Whenever variables outside of these specific areas were also included, they were usually identified as confounders or controls, moderators or mediators.

We visualized the combinations in which the different determinants have been studied in Fig 2 , which was obtained via multidimensional scaling and a subsequent cluster analysis (details outlined in S2 Appendix ). It depicts the spatial positioning of each determinant relative to all others, based on the number of times the effects of each pair of determinants have been studied simultaneously. When interpreting Fig 2 , one should keep in mind that determinants marked with an asterisk represent, in fact, collectives of variables.

Groups of determinants are marked by asterisks (see S1 Table in S1 Appendix ). Diminishing color intensity reflects a decrease in the total number of “connections” for a given determinant. Noteworthy pairwise “connections” are emphasized via lines (solid-dashed-dotted indicates decreasing frequency). Grey contour lines encircle groups of variables that were identified via cluster analysis. Abbreviations: age = population age distribution, associations = membership in associations, AT-index = atherogenic-thrombogenic index, BR = birth rate, CAPB = Cyclically Adjusted Primary Balance, civilian-labor = civilian labor force, C-section = Cesarean delivery rate, credit-info = depth of credit information, dissatisf = residents’ dissatisfaction, distrib.orient = distributional orientation, EDU = education, eHealth = eHealth index at GP-level, exch.rate = exchange rate, fat = fat consumption, GDP = gross domestic product, GFCF = Gross Fixed Capital Formation/Creation, GH-gas = greenhouse gas, GII = gender inequality index, gov = governance index, gov.revenue = government revenues, HC-coverage = healthcare coverage, HE = health(care) expenditure, HHconsump = household consumption, hosp.beds = hospital beds, hosp.payment = hospital payment scheme, hosp.stay = length of hospital stay, IDI = ICT development index, inc.ineq = income inequality, industry-labor = industrial labor force, infant-sex = infant sex ratio, labor-product = labor production, LBW = low birth weight, leave = work leave, life-satisf = life satisfaction, M-age = maternal age, marginal-tax = marginal tax rate, MDs = physicians, mult.preg = multiple pregnancy, NHS = Nation Health System, NO = nitrous oxide emissions, PM10 = particulate matter (PM10) emissions, pop = population size, pop.density = population density, pre-term = pre-term birth rate, prison = prison population, researchE = research&development expenditure, school.ref = compulsory schooling reform, smoke-free = smoke-free places, SO = sulfur oxide emissions, soc.E = social expenditure, soc.workers = social workers, sugar = sugar consumption, terror = terrorism, union = union density, UR = unemployment rate, urban = urbanization, veg-fr = vegetable-and-fruit consumption, welfare = welfare regime, Wwater = wastewater treatment.

https://doi.org/10.1371/journal.pone.0239031.g002

Distances between determinants in Fig 2 are indicative of determinants’ “connectedness” with each other. While the statistical procedure called for higher dimensionality of the model, for demonstration purposes we show here a two-dimensional solution. This simplification unfortunately comes with a caveat. To use the factor smoking as an example, it would appear it stands at a much greater distance from GDP than it does from alcohol. In reality however, smoking was considered together with alcohol consumption [ 21 , 25 , 26 , 52 , 68 ] in just as many studies as it was with GDP [ 21 , 25 , 26 , 52 , 59 ], five. To aid with respect to this apparent shortcoming, we have emphasized the strongest pairwise links. Solid lines connect GDP with health expenditure (HE), unemployment rate (UR), and education (EDU), indicating that the effect of GDP on health, taking into account the effects of the other three determinants as well, was evaluated in between 12 to 16 studies of the 60 included in this review. Tracing the dashed lines, we can also tell that GDP appeared jointly with income inequality, and HE together with either EDU or UR, in anywhere between 8 to 10 of our selected studies. Finally, some weaker but still worth-mentioning “connections” between variables are displayed as well via the dotted lines.

The fact that all notable pairwise “connections” are concentrated within a relatively small region of the plot may be interpreted as low overall “connectedness” among the health indicators studied. GDP is the most widely investigated determinant in relation to general population health. Its total number of “connections” is disproportionately high (159) compared to its runner-up–HE (with 113 “connections”), and then subsequently EDU (with 90) and UR (with 86). In fact, all of these determinants could be thought of as outliers, given that none of the remaining factors have a total count of pairings above 52. This decrease in individual determinants’ overall “connectedness” can be tracked on the graph via the change of color intensity as we move outwards from the symbolic center of GDP and its closest “co-determinants”, to finally reach the other extreme of the ten indicators (welfare regime, household consumption, compulsory school reform, life satisfaction, government revenues, literacy, research expenditure, multiple pregnancy, Cyclically Adjusted Primary Balance, and residents’ dissatisfaction; in white) the effects on health of which were only studied in isolation.

Lastly, we point to the few small but stable clusters of covariates encircled by the grey bubbles on Fig 2 . These groups of determinants were identified as “close” by both statistical procedures used for the production of the graph (see details in S2 Appendix ).

Statistical methodology

There was great variation in the level of statistical detail reported. Some authors provided too vague a description of their analytical approach, necessitating some inference in this section.

The issue of missing data is a challenging reality in this field of research, but few of the studies under review (12/60) explain how they dealt with it. Among the ones that do, three general approaches to handling missingness can be identified, listed in increasing level of sophistication: case-wise deletion, i.e., removal of countries from the sample [ 20 , 45 , 48 , 58 , 59 ], (linear) interpolation [ 28 , 30 , 34 , 58 , 59 , 63 ], and multiple imputation [ 26 , 41 , 52 ].

Correlations, Pearson, Spearman, or unspecified, were the only technique applied with respect to the health outcomes of interest in eight analyses [ 33 , 42 – 44 , 46 , 53 , 57 , 61 ]. Among the more advanced statistical methods, the family of regression models proved to be, by and large, predominant. Before examining this closer, we note the techniques that were, in a way, “unique” within this selection of studies: meta-analyses were performed (random and fixed effects, respectively) on the reduced form and 2-sample two stage least squares (2SLS) estimations done within countries [ 39 ]; difference-in-difference (DiD) analysis was applied in one case [ 23 ]; dynamic time-series methods, among which co-integration, impulse-response function (IRF), and panel vector autoregressive (VAR) modeling, were utilized in one study [ 80 ]; longitudinal generalized estimating equation (GEE) models were developed on two occasions [ 70 , 78 ]; hierarchical Bayesian spatial models [ 51 ] and special autoregressive regression [ 62 ] were also implemented.

Purely cross-sectional data analyses were performed in eight studies [ 25 , 45 , 47 , 50 , 55 , 56 , 67 , 71 ]. These consisted of linear regression (assumed ordinary least squares (OLS)), generalized least squares (GLS) regression, and multilevel analyses. However, six other studies that used longitudinal data in fact had a cross-sectional design, through which they applied regression at multiple time-points separately [ 27 , 29 , 36 , 48 , 68 , 72 ].

Apart from these “multi-point cross-sectional studies”, some other simplistic approaches to longitudinal data analysis were found, involving calculating and regressing 3-year averages of both the response and the predictor variables [ 54 ], taking the average of a few data-points (i.e., survey waves) [ 56 ] or using difference scores over 10-year [ 19 , 29 ] or unspecified time intervals [ 40 , 55 ].

Moving further in the direction of more sensible longitudinal data usage, we turn to the methods widely known among (health) economists as “panel data analysis” or “panel regression”. Most often seen were models with fixed effects for country/region and sometimes also time-point (occasionally including a country-specific trend as well), with robust standard errors for the parameter estimates to take into account correlations among clustered observations [ 20 , 21 , 24 , 28 , 30 , 32 , 34 , 37 , 38 , 41 , 52 , 59 , 60 , 63 , 66 , 69 , 73 , 79 , 81 , 82 ]. The Hausman test [ 83 ] was sometimes mentioned as the tool used to decide between fixed and random effects [ 26 , 49 , 63 , 66 , 73 , 82 ]. A few studies considered the latter more appropriate for their particular analyses, with some further specifying that (feasible) GLS estimation was employed [ 26 , 34 , 49 , 58 , 60 , 73 ]. Apart from these two types of models, the first differences method was encountered once as well [ 31 ]. Across all, the error terms were sometimes assumed to come from a first-order autoregressive process (AR(1)), i.e., they were allowed to be serially correlated [ 20 , 30 , 38 , 58 – 60 , 73 ], and lags of (typically) predictor variables were included in the model specification, too [ 20 , 21 , 37 , 38 , 48 , 69 , 81 ]. Lastly, a somewhat different approach to longitudinal data analysis was undertaken in four studies [ 22 , 35 , 48 , 65 ] in which multilevel–linear or Poisson–models were developed.

Regardless of the exact techniques used, most studies included in this review presented multiple model applications within their main analysis. None attempted to formally compare models in order to identify the “best”, even if goodness-of-fit statistics were occasionally reported. As indicated above, many studies investigated women’s and men’s health separately [ 19 , 21 , 22 , 27 – 29 , 31 , 33 , 35 , 36 , 38 , 39 , 45 , 50 , 51 , 64 , 65 , 69 , 82 ], and covariates were often tested one at a time, including other covariates only incrementally [ 20 , 25 , 28 , 36 , 40 , 50 , 55 , 67 , 73 ]. Furthermore, there were a few instances where analyses within countries were performed as well [ 32 , 39 , 51 ] or where the full time period of interest was divided into a few sub-periods [ 24 , 26 , 28 , 31 ]. There were also cases where different statistical techniques were applied in parallel [ 29 , 55 , 60 , 66 , 69 , 73 , 82 ], sometimes as a form of sensitivity analysis [ 24 , 26 , 30 , 58 , 73 ]. However, the most common approach to sensitivity analysis was to re-run models with somewhat different samples [ 39 , 50 , 59 , 67 , 69 , 80 , 82 ]. Other strategies included different categorization of variables or adding (more/other) controls [ 21 , 23 , 25 , 28 , 37 , 50 , 63 , 69 ], using an alternative main covariate measure [ 59 , 82 ], including lags for predictors or outcomes [ 28 , 30 , 58 , 63 , 65 , 79 ], using weights [ 24 , 67 ] or alternative data sources [ 37 , 69 ], or using non-imputed data [ 41 ].

As the methods and not the findings are the main focus of the current review, and because generic checklists cannot discern the underlying quality in this application field (see also below), we opted to pool all reported findings together, regardless of individual study characteristics or particular outcome(s) used, and speak generally of positive and negative effects on health. For this summary we have adopted the 0.05-significance level and only considered results from multivariate analyses. Strictly birth-related factors are omitted since these potentially only relate to the group of infant mortality indicators and not to any of the other general population health measures.

Starting with the determinants most often studied, higher GDP levels [ 21 , 26 , 27 , 29 , 30 , 32 , 43 , 48 , 52 , 58 , 60 , 66 , 67 , 73 , 79 , 81 , 82 ], higher health [ 21 , 37 , 47 , 49 , 52 , 58 , 59 , 68 , 72 , 82 ] and social [ 20 , 21 , 26 , 38 , 79 ] expenditures, higher education [ 26 , 39 , 52 , 62 , 72 , 73 ], lower unemployment [ 60 , 61 , 66 ], and lower income inequality [ 30 , 42 , 53 , 55 , 73 ] were found to be significantly associated with better population health on a number of occasions. In addition to that, there was also some evidence that democracy [ 36 ] and freedom [ 50 ], higher work compensation [ 43 , 79 ], distributional orientation [ 54 ], cigarette prices [ 63 ], gross national income [ 22 , 72 ], labor productivity [ 26 ], exchange rates [ 32 ], marginal tax rates [ 79 ], vaccination rates [ 52 ], total fertility [ 59 , 66 ], fruit and vegetable [ 68 ], fat [ 52 ] and sugar consumption [ 52 ], as well as bigger depth of credit information [ 22 ] and percentage of civilian labor force [ 79 ], longer work leaves [ 41 , 58 ], more physicians [ 37 , 52 , 72 ], nurses [ 72 ], and hospital beds [ 79 , 82 ], and also membership in associations, perceived corruption and societal trust [ 48 ] were beneficial to health. Higher nitrous oxide (NO) levels [ 52 ], longer average hospital stay [ 48 ], deprivation [ 51 ], dissatisfaction with healthcare and the social environment [ 56 ], corruption [ 40 , 50 ], smoking [ 19 , 26 , 52 , 68 ], alcohol consumption [ 26 , 52 , 68 ] and illegal drug use [ 68 ], poverty [ 64 ], higher percentage of industrial workers [ 26 ], Gross Fixed Capital creation [ 66 ] and older population [ 38 , 66 , 79 ], gender inequality [ 22 ], and fertility [ 26 , 66 ] were detrimental.

It is important to point out that the above-mentioned effects could not be considered stable either across or within studies. Very often, statistical significance of a given covariate fluctuated between the different model specifications tried out within the same study [ 20 , 49 , 59 , 66 , 68 , 69 , 73 , 80 , 82 ], testifying to the importance of control variables and multivariate research (i.e., analyzing multiple independent variables simultaneously) in general. Furthermore, conflicting results were observed even with regards to the “core” determinants given special attention, so to speak, throughout this text. Thus, some studies reported negative effects of health expenditure [ 32 , 82 ], social expenditure [ 58 ], GDP [ 49 , 66 ], and education [ 82 ], and positive effects of income inequality [ 82 ] and unemployment [ 24 , 31 , 32 , 52 , 66 , 68 ]. Interestingly, one study [ 34 ] differentiated between temporary and long-term effects of GDP and unemployment, alluding to possibly much greater complexity of the association with health. It is also worth noting that some gender differences were found, with determinants being more influential for males than for females, or only having statistically significant effects for male health [ 19 , 21 , 28 , 34 , 36 , 37 , 39 , 64 , 65 , 69 ].

The purpose of this scoping review was to examine recent quantitative work on the topic of multi-country analyses of determinants of population health in high-income countries.

Measuring population health via relatively simple mortality-based indicators still seems to be the state of the art. What is more, these indicators are routinely considered one at a time, instead of, for example, employing existing statistical procedures to devise a more general, composite, index of population health, or using some of the established indices, such as disability-adjusted life expectancy (DALE) or quality-adjusted life expectancy (QALE). Although strong arguments for their wider use were already voiced decades ago [ 84 ], such summary measures surface only rarely in this research field.

On a related note, the greater data availability and accessibility that we enjoy today does not automatically equate to data quality. Nonetheless, this is routinely assumed in aggregate level studies. We almost never encountered a discussion on the topic. The non-mundane issue of data missingness, too, goes largely underappreciated. With all recent methodological advancements in this area [ 85 – 88 ], there is no excuse for ignorance; and still, too few of the reviewed studies tackled the matter in any adequate fashion.

Much optimism can be gained considering the abundance of different determinants that have attracted researchers’ attention in relation to population health. We took on a visual approach with regards to these determinants and presented a graph that links spatial distances between determinants with frequencies of being studies together. To facilitate interpretation, we grouped some variables, which resulted in some loss of finer detail. Nevertheless, the graph is helpful in exemplifying how many effects continue to be studied in a very limited context, if any. Since in reality no factor acts in isolation, this oversimplification practice threatens to render the whole exercise meaningless from the outset. The importance of multivariate analysis cannot be stressed enough. While there is no “best method” to be recommended and appropriate techniques vary according to the specifics of the research question and the characteristics of the data at hand [ 89 – 93 ], in the future, in addition to abandoning simplistic univariate approaches, we hope to see a shift from the currently dominating fixed effects to the more flexible random/mixed effects models [ 94 ], as well as wider application of more sophisticated methods, such as principle component regression, partial least squares, covariance structure models (e.g., structural equations), canonical correlations, time-series, and generalized estimating equations.

Finally, there are some limitations of the current scoping review. We searched the two main databases for published research in medical and non-medical sciences (PubMed and Web of Science) since 2013, thus potentially excluding publications and reports that are not indexed in these databases, as well as older indexed publications. These choices were guided by our interest in the most recent (i.e., the current state-of-the-art) and arguably the highest-quality research (i.e., peer-reviewed articles, primarily in indexed non-predatory journals). Furthermore, despite holding a critical stance with regards to some aspects of how determinants-of-health research is currently conducted, we opted out of formally assessing the quality of the individual studies included. The reason for that is two-fold. On the one hand, we are unaware of the existence of a formal and standard tool for quality assessment of ecological designs. And on the other, we consider trying to score the quality of these diverse studies (in terms of regional setting, specific topic, outcome indices, and methodology) undesirable and misleading, particularly since we would sometimes have been rating the quality of only a (small) part of the original studies—the part that was relevant to our review’s goal.

Our aim was to investigate the current state of research on the very broad and general topic of population health, specifically, the way it has been examined in a multi-country context. We learned that data treatment and analytical approach were, in the majority of these recent studies, ill-equipped or insufficiently transparent to provide clarity regarding the underlying mechanisms of population health in high-income countries. Whether due to methodological shortcomings or the inherent complexity of the topic, research so far fails to provide any definitive answers. It is our sincere belief that with the application of more advanced analytical techniques this continuous quest could come to fruition sooner.

Supporting information

S1 checklist. preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (prisma-scr) checklist..

https://doi.org/10.1371/journal.pone.0239031.s001

S1 Appendix.

https://doi.org/10.1371/journal.pone.0239031.s002

S2 Appendix.

https://doi.org/10.1371/journal.pone.0239031.s003

• View Article
• Google Scholar
• PubMed/NCBI
• 75. Dahlgren G, Whitehead M. Policies and Strategies to Promote Equity in Health. Stockholm, Sweden: Institute for Future Studies; 1991.
• 76. Brunner E, Marmot M. Social Organization, Stress, and Health. In: Marmot M, Wilkinson RG, editors. Social Determinants of Health. Oxford, England: Oxford University Press; 1999.
• 77. Najman JM. A General Model of the Social Origins of Health and Well-being. In: Eckersley R, Dixon J, Douglas B, editors. The Social Origins of Health and Well-being. Cambridge, England: Cambridge University Press; 2001.
• 85. Carpenter JR, Kenward MG. Multiple Imputation and its Application. New York: John Wiley & Sons; 2013.
• 86. Molenberghs G, Fitzmaurice G, Kenward MG, Verbeke G, Tsiatis AA. Handbook of Missing Data Methodology. Boca Raton: Chapman & Hall/CRC; 2014.
• 87. van Buuren S. Flexible Imputation of Missing Data. 2nd ed. Boca Raton: Chapman & Hall/CRC; 2018.
• 88. Enders CK. Applied Missing Data Analysis. New York: Guilford; 2010.
• 89. Shayle R. Searle GC, Charles E. McCulloch. Variance Components: John Wiley & Sons, Inc.; 1992.
• 90. Agresti A. Foundations of Linear and Generalized Linear Models. Hoboken, New Jersey: John Wiley & Sons Inc.; 2015.
• 91. Leyland A. H. (Editor) HGE. Multilevel Modelling of Health Statistics: John Wiley & Sons Inc; 2001.
• 92. Garrett Fitzmaurice MD, Geert Verbeke, Geert Molenberghs. Longitudinal Data Analysis. New York: Chapman and Hall/CRC; 2008.
• 93. Wolfgang Karl Härdle LS. Applied Multivariate Statistical Analysis. Berlin, Heidelberg: Springer; 2015.

• Search This Site All UCSD Sites Faculty/Staff Search Term
• At A Glance
• Dean's Message
• Annual Report
• Bachelor of Science in Public Health
• Master of Public Health
• Master of Science in Biostatistics
• Master of Advanced Studies, Leadership of Healthcare Organizations
• Joint Doctoral Program in Public Health
• Doctoral Program in Biostatistics
• NIH T32 Programs
• UC San Diego - San Diego State University General Preventive Medicine Residency
• Faculty Mentor Training Program
• Biostatistics & Bioinformatics
• Climate and Environmental Sciences
• Community Health Services and Preventive Medicine
• Epidemiology
• Global Health
• Health Behavior
• Health Policy
• Public Mental Health and Substance Use
• Technology and Precision Health
• Climate and Environmental Health
• Health Equity and Global Health Justice
• Health Services Research and Health Policy
• Healthy Aging and Human Longevity Science
• Mental Health and Substance Use
• Quantitative Methods in Public Health
• Women's Health and Reproductive Justice
• Research Partners
• Research Recharge Cores
• Faculty Research Labs
• Administration
• Academics and MSP Physicians
• Benefactors
• Recruitment
• Available Research Talent
• Commencement 2024
• COVID-19 In the News

Our Research

The scope of Quantitative Methods in Public Health is broad, ranging from biostatistics to bioinformatics to biomedical data science, as well as experimental design, and other quantitative methods as applied to public health and biomedicine in general. We aim to become a hub of these focuses on the UC San Diego campus. 

Ronghui (Lily) Xu, PhD, Professor  - Program Lead

Academic members: Gretchen Bandoli, PhD, Assistant Professor, Secondary Appointed Cinnamon Bloss, PhD, Professor and Assistant Dean for Academic Affairs Brian Chen, PhD, MPH, Assistant Professor Joachim Ix, MD, MAS, Professor, Secondary Appointed Sonia Jain, PhD, Msc, Professor and Associate Dean for Justice, Equity, Diversity and Inclusion Jordan Kohn, Assistant Project Scientist Lin Liu, PhD, Associate Professor Loki Natarajan, PhD, Professor and Associate Dean for Research Corinne Peek-Asa, PhD, MPH, Distinguished Professor and Vice Chancellor for Research  Rany Salem, PhD, MPH, Assistant Professor Armin Schwartzman, PhD, Professor Matthew Stone, PhD, Assistant Professor Xin Tu, PhD, Professor Florin Vaida, PhD, Professor Jill Waalen, MD, PhD, Associate Diplomat Xinlian Zhang, PhD, Assistant Professor Jingjing Zou, PhD, Assistant Professor

Staff members: Maliha Safdar

Research Labs

Dr. Xin Tu  is a Professor of Biostatistics in the  Division of Biostatistics and Bioinformatics . His group specializes in semiparametric and machine learning models and their applications to mental health and psychosocial research, and has extensive collaborations with investigators and multiple centers at UC San Diego.  The group’s statistical methodologic research focuses on models for between-subject attributes, a new paradigm for facilitating infererence for classic statistical models for within-subject attributes, as well as modeling emerging high-dimensional data arising in biomedical and network research. 

Public Health Institute

Search Search

Explore Our Work

Work With Us

• Thought Leadership
• Meet Our Experts
• Press & Updates
• Policy & Advocacy

Focus Areas

• Alcohol, Tobacco, Drugs & Mental Health
• Capacity Building & Leadership
• Chronic Disease Prevention
• Communicable Disease Prevention
• Data, Technology & Innovation
• Disability Rights
• Environmental Health
• Global Health
• Health Care & Population Health
• Healthy Communities
• Women, Youth & Children
• All PHI Programs

Bring Your Work to PHI

• Partner with PHI
• Inquire About Bringing Your Work or Partnering with PHI
• RFP & Vendor Opportunities
• Non-PHI Opportunities with PHI Programs: Americorps Vista
• Current Opportunities
• PHI REDI Statement
• 2022 Annual Report
• Leadership & Board
• Financials & Compliance

Research – Quantitative

To make an effective impact on community health, public health practices, programs and policies must be informed by accurate research and analysis. The Public Health Institute works to foster health, well-being, and quality of life through rigorous quantitative research guided by the principles of health equity, environment, education, and economic contexts. By exploring statistical associations between variables and finding differing patterns in population health outcomes, our programs help to advance research and practice, evaluate the cost of health inequities and disease, design successful programs and interventions, and improve health outcomes in communities.

Our Expertise Can Make Your Work Stronger

See how PHI can work with you to support or lead your health initiatives.

See all Research – Quantitative Impacts

• 53M adults harmed by another's drinking
• 4x the breast cancer cases among DDT exposed women
• 3 generations of women including in the CHDS research

Providing Research and Tools Local Health Coalitions Can Use to Measure their Progress in Collaborating for Community Health

Reaching 1.1M Through SNAP-Ed, Training, Partnerships and Research to Address Food Insecurity

Wildfires and Best Practices for Sharing Household Water Contamination Test Results

Developing Best Practices in Sharing Study Findings with Participants

Redefining Alcohol Use Disorder to Improve Diagnosis and Treatment

Connecting Moderate Drinking to the Risk of Injuries

PHI’s CHDS Produces Special Issue of “Reproductive Toxicology” on Pregnancy Exposure and Breast Cancer

Understanding the Roots of Autism

Pioneering Research on the Impacts of Generational Exposure to DDT & PFASs

Exposing the Hidden Health Threat of Undiagnosed Childhood Lead Poisoning

Calculating the Costs of Environmental Health Conditions in California Children

Advancing 60+ Years of Research with PHI’s Alcohol Research Group

Using Real-Time Air Monitoring to Reduce Pollution Risk

Identifying Connections Between Women’s Drinking and Diabetes Risk

Finding Connections Between Cancer Diagnoses and Drinking Behaviors

Helping Programs & Researchers Thrive With Strong Grant Support

Understanding the Population Risks of PBDEs

Active programs, alcohol research group, behavioral health and recovery studies, center for connected health policy, child health and development studies, getting it right from the start: regulation of recreational marijuana, national alcohol research center, archived programs.

PHI is Leading the Conversation

Study Finds Children with Sickle Cell Anemia May Not Receive Antibiotics, Stroke Screenings

Hidden Inequities: Intersectional Study Uncovers Stark Disparities in Substance Use Treatment Completion Among Women of Color

Patients Given Buprenorphine in Emergency Departments More Likely to Have Sustained Treatment for Opioid Use Disorders

Phi experts working in this area:.

Andrew Herring, MD

Program Director, Bridge CA Bridge

Alain Monnereau, MD, PhD

Research Program Director; SEER Principal Investigator, Cancer Registry of Greater California

Mohammad Sediq Hazratzai, MD, MPH

Program Director, Sehat Initiative

Lynn Silver, MD, MPH, FAAP

Program Director, Safely Opening Schools Lynn Silver, Researcher California Alliance for Prevention Funding

Barbara Cohn, PhD, MPH, AB

Director and Senior Research Scientist, Child Health and Development Studies

Doug Polcin, EdD, MFT

Program Director, Behavioral Health and Recovery Studies

Robert Harrison, MD, MPH

Chief, California Occupational Health Surveillance and Evaluation Program (OHSEP)

Christy McCain, MPH

Research Scientist, CA-Pregnancy-Associated Mortality Review

Bethany Young Holt, PhD, MPH

Executive Director, CAMI Health | Director, Initiative for IMPTs, CAMI Health

You change the world. We do the rest. Explore fiscal sponsorship at PHI.

Together, we can accelerate our response to public health’s most critical issues.

Find Employment

Begin your career at the Public Health Institute.

Wildfires & Extreme Heat: Resources to Protect Yourself & Your Community

Communities across the U.S. and around the world are grappling with dangerous wildfires and extreme heat. These threats disrupt and uproot communities and pose serious risks to environmental and community health—from rising temperatures, unhealthy air pollutants, water contamination and more. Find PHI tools, resources and examples to help communities take action and promote climate safety, equity and resiliency.

Get started

Continue to PHI.org

Cookies on GOV.UK

We use some essential cookies to make this website work.

We’d like to set additional cookies to understand how you use GOV.UK, remember your settings and improve government services.

We also use cookies set by other sites to help us deliver content from their services.

You have accepted additional cookies. You can change your cookie settings at any time.

You have rejected additional cookies. You can change your cookie settings at any time.

• Health and social care
• Public health

Evaluation methods: evaluation in health and wellbeing

Helping public health practitioners conducting evaluations – choosing evaluation methods.

Choosing methods for evaluation

A wide variety of research methods and data collection tools are available for use in evaluation: qualitative and quantitative. Different methods are suitable for answering different types of evaluation questions. This section provides an overview of the most common methods. The resources referred to at the end of this section give more detail on different methods and how to use them.

Qualitative research

Qualitative research encompasses a variety of methods, but is often defined in terms of using words and text in data collection and analysis, rather than using numerical measurement and quantification (Bryman, A (2015), ‘Social research methods’ (fifth edition) Oxford University Press, Oxford). It commonly investigates how people make sense of the world and experience events. For example, it might explore how it feels to be a patient with a long-term condition or what it is like to live with someone with a terminal illness, and aim to gain insight into how people make sense of and manage these situations.

Qualitative research is usually ‘bottom up’ rather than ‘top down’. That is, a theory or explanation is developed from the data rather than data being collected to test a theory or hypothesis. Qualitative research often aims to describe and explain rather than predict or identify cause and effect. It can be used to enrich our understanding of an issue or to improve services (Willig, C (2008), ‘Introducing qualitative research in psychology: adventures in theory and method’ Open University Press, Oxford).

Qualitative research is not concerned with representing a population, in the way that quantitative research is, but with investigating topics in depth (Willig, 2008). Usually, a relatively small number of people are included in a sample because of the richness of the data collected and the time it takes to collect and analyse information. Sometimes, simple qualitative research methods are combined with quantitative research, for example, via the inclusion of a comments section within a questionnaire.

Benefits of qualitative research

The benefits to using qualitative methods include that they:

• are useful for investigating the perspectives and interpretations of participants in a holistic fashion
• are less dependent than quantitative methods on the pre-conceived ideas of the researchers
• facilitate new or deeper insights into, and understanding of, a phenomenon (this can be very useful where little is known about a topic and exploratory research is required)
• can be used to investigate unusual cases from which valuable insights can be gained (in quantitative research these ‘outliers’ may be discarded)
• can usually be modified on an ongoing basis as new insights are gained – for example, to explore if unexpected findings emerge
• is adaptable – for example, the order and style of questions in an interview or focus group can be altered to fit or accommodate the needs of different participants or groups
• can be coded and summarised in a quantitative way, if desired

Limitations of qualitative research

The limitations of qualitative methods include:

• they are generally time-consuming to use – it can take a lot of time to arrange and undertake data collection (such as via interviews and focus groups), transcribe discussions, analyse data, and interpret and present findings in a meaningful way
• data analysis can be challenging and relies on having the necessary knowledge and skills, such as in qualitative data analysis approaches, techniques, analysis software
• they produce data and findings which may vary depending on the skills and theoretical perspectives of the researchers collecting, analysing and interpreting the data – however, the reporting of qualitative research should make this process transparent to allow the reader to judge findings in light of this
• they usually collect and analyse data from relatively small numbers of participants, meaning that the generalisability of findings is not addressed and trends or distributions of views within a population are not ascertained
• the anonymity and confidentiality of participants is more difficult to maintain

Qualitative research tools

Interviews are frequently used to collect data in qualitative research. They are usually based on a topic guide. In evaluation studies, interviews are commonly semi-structured where questions or themes are decided in advance, but the interviewer has flexibility to re-order the questions and follow-up with further questions, if necessary.

Interviews allow topics to be explored in depth. These topics can be adapted to take account of participant’s needs (for example, if there are language or disability challenges) or experiences (for example, skipping questions which are not relevant). An interviewee might also discuss a topic without being prompted or the interviewer might change the order of the questions depending on the situation.

In semi-structured interviews, questions are usually relatively ‘open’ – that is, they encourage a detailed answer from the interviewee rather than a ‘yes’ or ‘no’ response. Generally, the interview topic guide or schedule would contain prompts in case a topic doesn’t naturally arise. Leading questions or otherwise indicating to an interviewee that a certain type of answer is the ‘right’ one should be avoided as this would bias the data.

Interviews can be conducted by telephone, through an online video resource or in person. There are benefits and drawbacks of using the telephone or online video compared to face-to-face interviews, summarised below.

Advantages of telephone or online video interviews are they:

• are cheaper in terms of staff time and travel
• avoid wasted time if the sample population has a high rate of ‘no-shows’ for interviews
• are more practical if the interviewee lives far away, or if they have problems travelling
• allow more people to be interviewed due to reduced cost and time

Limitations of telephone or online video interviews are they:

• make it more difficult to establish a rapport with the interviewee
• make it difficult to assess how interviewees are reacting to questions because their body language cannot be observed
• tend to be shorter and are less suitable for sensitive topics
• rely on having access to online video packages or good phone connection or reception, which may be problematic for some harder-to-reach participants

Advantages of face-to-face interviews are they:

• capture verbal and non-verbal cues and responses to questions
• observe emotions and body language
• establish a rapport with the interviewee

Limitations of face-to-face video interviews are:

• they require a number of people to conduct the interviews, which means there will be personnel costs
• quality of data you receive will often depend on the ability of the interviewer
• size of the sample can be limited to the size of your interviewing staff, the area in which the interviews are conducted, and the number of qualified respondents within that area
• they are more expensive and require time for travelling

In advance of the interview, researchers should decide if: the interview should be recorded, how it will be transcribed and analysed and how excerpts will be presented in any write up. In line with research ethics, it is good practice to discuss confidentiality issues and ask interviewees to read and sign a consent form to confirm that they are happy with the arrangements. Complete transcription of recordings is very time-consuming and partial transcriptions or summaries are sometimes used, depending on the type of evaluation.

Interviews can be tiring for interviewers, as they have to listen carefully to determine which topic to go to next and to ensure understanding. If the interviews are lengthy, it is generally a good idea not to arrange more than 2 in a day.

Focus groups

Focus groups bring together a small number of selected participants to discuss a specific topic and are usually facilitated by someone who does not know the group. The participants might be strangers or people who meet together already. In general, it is best not to include people with more power than others in the same group, such as teachers and pupils or junior and senior staff, as this can stop participants being open with their views or opinions. It takes time to set up a focus group and to ensure that everyone attends. Set up can be easier if the group is run alongside a regular meeting. For example, if you are interested in the views of men with an alcohol problem, you might try to get permission to conduct a focus group before or after an alcohol support group session.

Focus groups have similarities to interviews in that a facilitator uses a topic guide or scenario (see below) to ask particular questions and steer the discussion. The person who facilitates the focus group should be experienced, and there is training available to support development of relevant skills. Facilitators must ensure that one person does not dominate the discussion and that everyone gets the opportunity to have their views heard. The facilitator should not have a particular vested interest in the findings, so they do not lead the discussion or impose their own views. The facilitator should always be neutral.

It is useful to record a focus group or at least have a second person taking notes of the discussion as it is difficult to do this alongside facilitating the group. As with interviews, issues of consent and confidentiality should be discussed with participants.

Advantages of focus groups are that they:

• are useful when the interactions and contributions within the group may prompt additional, interesting themes to arise
• are useful when time is short and there is a need to get a range of people’s views quickly
• tend to work well for existing groups where people are comfortable with speaking to each other and are confident in expressing their opinion in a group situation
• can be useful for understanding how and why different groups or types of participants have differing views (for example, men versus women, patients versus health professionals)

Disadvantages of focus groups are that:

• they are generally less useful for recording the responses of individual participants, as it can be difficult to note this down accurately during a discussion, or to identify individuals from a recording during transcription
• they are less suitable to use with some participants – especially if the topic to be discussed is sensitive
• people with hearing problems might also find a group discussion difficult
• they can be challenging and require training to facilitate, especially if there are disagreements amongst participants, dominant participants, or discussions get side-tracked away from the main topics of interest
• they produce complex data which can be particularly difficult and time-consuming to transcribe, summarise, and analyse

Topic guide

Topic guides include questions and prompts used when conducting a focus group or interview. They are called guides because they are meant to be adapted to the needs and experiences of the interviewee or focus group and used in a flexible way. The content of a guide would be decided based on the evaluation questions and objectives as well as after discussions with any oversight or steering group and/or with the type of people you will be interviewing. It is a good idea to test questions with a pilot group to ensure they are understandable and appropriate for the type of participants in the sample.

For example, if you were interviewing smokers about resources and services for quitting smoking in Manchester, you may wish to meet with a group of smokers in Liverpool (to talk to smokers in a similar area who are likely to have similar experiences) to discuss what topics should be included and what language should be used. It is most useful to ask open-ended questions to encourage participants to speak, and it is usual to start with some generic questions at the beginning to relax the participants and build a rapport.

For example, you might want to start with a question which asks for some background and gauges the participants’ knowledge of the area. This will help with adapting later topics and deciding in which order to ask questions. The opening questions also give insight into how forthcoming participants are likely to be. Some people speak readily and at length whereas others can be more reluctant to express their views and need prompts to explore issues in depth. This is particularly important if the topic is of a sensitive nature.

Scenarios can also be used in interviews or focus groups to depersonalise an issue. If a topic is sensitive and people might be uncomfortable talking about their own experiences, they could be asked to give advice to someone else in a similar situation or to imagine a person and what their responses would be.

For example, if you wanted to interview someone about addictions and their views on the services available, you could ask: ‘If a friend or neighbour wished to access addiction services in this area what advice would you give them? What problems might you highlight? Where would you suggest that they start?’

Alternatively, if you wanted to gain insight into the concerns of someone with terminal illness regarding hospices, you might say: ‘Mr McKenzie is moving to a hospice 20 miles from his home after a long illness. His wife is elderly and 2 of his 4 children have moved away from the area. What do you think his concerns might be? What about those of his wife and children?’

This would allow participants to discuss a topic in a more general way without feeling that the questions are intrusive or require sharing too much personal information.

Other qualitative methods

There are many more qualitative research methods such as naturalistic or ethnographic observations, video recordings of behaviour, open-ended questions on questionnaires, and analysis of text (for example, in advertising or educational materials) or photographs.

Analysing qualitative data

In-depth qualitative analysis requires training and experience. It can be conducted in different ways but usually proceeds in the following stages.

• It is usually helpful to thoroughly familiarise yourself with the data by reading and re-reading transcripts or notes, or by listening to recordings.
• Next, segments of data are labelled through ‘coding’, and then higher level, more interpretive themes are developed from these codes in one or more further stages.
• Later stages often involve organising themes (for example, into groups), exploring their relationships with each other and observing patterns in the data.

In this way, qualitative analysis goes beyond merely summarising data, to drawing out underlying themes that provide in-depth insight into an issue.

There are computer packages that can help with analysis of qualitative data. The most common of these are NVivo, Atlas and MAXQDA. These programmes allow large amounts of qualitative data to be stored, analysed and summarised in a systematic way. Use of these packages requires training and a good understanding of qualitative methods.

Quantitative methods

Quantitative methods are used to investigate things that can be measured or quantified to generate numerical data. For example, what percentage of people hold a particular view, does use of a service vary by gender and age, what has changed after a particular intervention? Quantitative research aims to be objective and usually collects data in a pre-defined, structured way (Black, T R (1999), ‘Doing quantitative research in the social sciences: an integrated approach to research design, measurement and statistics’ Sage Publishing). It is used to describe general trends or distributions, or to test a theory or hypothesis about relationships between things, including cause and effect.

Quantitative research methods usually aim to tell you something about a population (for example, smokers or older adults) based on a representative sample selected for a study. However, evaluation studies do not always make claims for the population at large, and may only provide more specific information about the effects of an intervention in a specific group of people or context.

Often, the aim of an evaluation is to find out how things have changed after an intervention. For example, are clinicians more knowledgeable about correct hand washing techniques after a training session? Alternatively, an evaluation may assess whether a service achieves a pre-existing standard. For example, have young people with mental health problems been offered support within a defined time period? See the section on outcome evaluation.

When planning an evaluation, you should identify what data is needed, from whom and when. This will be influenced by the questions being addressed and the most suitable study design for answering these questions within the resources available. It is important to determine if the required data are already available and accessible, or whether the data will need to be collected as this will have time and resource implications. It may be possible to directly observe (for example, to produce counts) or measure (for example, using specialist equipment or monitoring to assess physiological markers) the phenomena of interest. However, evaluation often relies on self-reported or pre-existing data, and various tools are available to support collection of this type of data. See the section on evaluation planning.

Quantitative research tools

Surveys and questionnaires.

A common and quick way of collecting quantitative data from large samples of people is through surveys and questionnaires. Data collected from questionnaires and surveys allow comparisons between groups and subgroups to decide if there are differences between them, and to explore potential reasons for these differences.

Questionnaires are less useful when you wish to conduct more exploratory research (for example, where you might not have enough understanding of something to know what questions to include). They are also unsuitable for understanding aspects such as the meaning of participant experiences or for exploring issues in-depth. In these cases, qualitative methods are usually more suitable.

Questionnaire administration

There are different ways of administering surveys and questionnaires. The advantages and limitations of some of the more common methods are outlined below.

Advantages of administering surveys in person:

• you can lead participant through appropriate sections, explaining what is required, if necessary
• high response rate and minimal errors or missing responses
• good for overcoming disability and language problems

Limitations of administering surveys in person:

• time consuming
• geographically limited
• problematic for sensitive topics

Advantages of administering surveys online:

• wide geographical reach
• can ‘force’ responses and lead people through questions to reduce missing data
• people can do in own time
• can ensure anonymity

Limitations of administering surveys online:

• response rate may be low
• internet access may be a problem and exclude certain groups (for example, those with sensory impairments or older people)
• requires technical knowledge to set up

Advantages of mailing out surveys:

Limitations of mailing out surveys:

• risk of very low response rates
• may need to build in time and budget for reminders or incentives
• need to be very clearly designed and self-explanatory

Designing a questionnaire

Questionnaires are usually accompanied by some demographic questions about age, gender, employment, place of residence and so on. For evaluations, specific questions are then based on what the intervention to be evaluated aims to achieve. If possible, it is better to use or adapt a questionnaire that has been used for similar purposes or in a similar sample and, ideally, has been demonstrated to be reliable and valid – see outcome evaluation section.

When selecting or designing a questionnaire, these issues may be considered:

• who is your sample; who do you need to answer the questions; will they complete the questionnaire
• where or how can you best reach these people
• how can you ensure a good response rate
• if the answers are confidential, how to ensure this
• are the questions simple and clear and only ask about one issue at a time
• avoid jargon in questions - this can be tested by piloting the questionnaire
• avoid leading questions (for example you would not ask ‘why did you like the service?’; ask ‘did you like the service? If so can you say why?’) – exclude any questions that are not necessary for the study
• exclude any questions to which the respondent might not know the answer
• include an ‘other’ option when necessary and provide space for respondents to write or add their own answers
• keep the questionnaire as short as possible
• leave any sensitive or confidential questions to the end to avoid putting people off at an early stage
• check whether the questionnaire is reliable and valid for the sample to be studied
• plan how the data generated by the questionnaire will be captured and stored (for example, entered into a spreadsheet), summarised (for example, to produce overall scores) and analysed

Secondary data collection

It is not always necessary to collect new data as data relevant to an evaluation may already exist – for example, data that’s been collected for administrative purposes or added to patient records. Certain permissions (for example ethics approval, letters of access) may need to be obtained to allow access to these data. A data collection ‘pro forma’ which specifies the data required is sometimes designed and can then be populated using the existing data sources. Alternatively, a request can be made for data or reports for a sample of patients to be transferred directly to the evaluator.

Secondary data collection, however, can also have its own challenges. Some data may be confidential or stored in a way that makes the data hard to access or to use for the desired purpose. Sometimes, data should be available in theory, but, in practice, they are missing, misfiled or not collected consistently.

Analysing quantitative data

Some quantitative analyses required for evaluation will be quite simple – for example, describing the number or percentage of people with a certain characteristic, achieving a goal or expressing a particular opinion. This analysis can usually be managed in Excel. More complex analyses (for example, comparing groups, looking at difference between subgroups, or controlling for extraneous factors) will require knowledge of statistical analyses and associated software packages (for example, SPSS, Stata, SAS). Support for statistical training or knowledge of the relevant software packages will need be to identified when planning an evaluation.

Combining qualitative and quantitative research

Qualitative and quantitative methods can usefully be combined in ‘mixed methods’ research to provide more comprehensive information for an evaluation (Andrew, S and Halcomb, E J (2009), ‘Mixed methods research for nursing and the health sciences’ Blackwell, West Sussex). You may wish to use qualitative research to find out how people feel and follow this with quantitative research to find out if these feelings are commonly experienced.

For example, if you want to explore pregnant mothers’ understanding and knowledge of antenatal care and their compliance with recommendations, you could start by conducting a focus group of mothers from different areas and backgrounds. This qualitative research would give you an insight into their knowledge and other the main topics that are important. You could then use these topics to develop a questionnaire to discover what is happening across the population of expecting mothers as a whole.

Alternatively, you may have obtained findings from a questionnaire and wish to discuss these with a focus group to get a deeper understanding of what the results mean. For example, if data from a survey or questionnaire shows that one hospital has more infections than others, you may wish to follow this up by interviewing staff or visiting the hospital and undertaking observations to get a more in-depth understanding of why that might be and how it could be resolved.

Andrew, S and Halcomb, E J (2009): ‘Mixed methods research for nursing and the health sciences’. Blackwell, West Sussex.

Black, T R (1999): ‘Doing quantitative research in the social sciences: an integrated approach to research design, measurement and statistics’. Sage Publishing.

Bryman, A (2015): ‘Social research methods’ (fifth edition). Oxford University Press, Oxford.

Guest, G. and Namely, E E (2014): ‘Public health research methods’. Sage Publishing.

Willig, C (2008): ‘Introducing qualitative research in psychology: adventures in theory and method’. Open University Press, Oxford.

Acknowledgements

Written by Sarah Denford, Jane Smith, Sarah Morgan Trimmer, Charles Abraham, and Krystal Warmoth, Psychology Applied to Health, University of Exeter Medical School.

This work was partially funded by the UK National Institute for Health Research ( NIHR ) School for Public Health Research, the NIHR Collaboration for Leadership in Applied Health Research and Care of the South West Peninsula (PenCLAHRC) and by Public Health England. However, the views expressed are those of the authors.

Related content

Is this page useful.

• Yes this page is useful
• No this page is not useful

Help us improve GOV.UK

Don’t include personal or financial information like your National Insurance number or credit card details.

To help us improve GOV.UK, we’d like to know more about your visit today. Please fill in this survey (opens in a new tab) .

Quantitative Methods

The Quantitative Methods (QM) field of study provides students with the neces­sary quantitative and analytical skills to approach and solve prob­lems in public health and clinical research and practice. This field is designed for mid-career health professionals, research scientists, and MD/MPH specific dual/joint-degree students.  

Through a competency-based curriculum, health professionals in the MPH-45 receive the analytical and statistical knowledge and skills required for successful public health prac­tice and research. In addition to providing broad perspectives on general aspects of public health, the QM field of study provides an excellent foundation for those interested in pursuing academic careers in the health sciences.  

Degree programs  

The Master of Public Health 45-credit degree provides established professionals with the specialized skills and powerful global network needed to progress their careers in public health.  

• Abbreviation: MPH-45 QM  
• Degree format: On campus  
• Time commitment: Full-time or part-time  
• Average program length: One year full-time; two years part-time  

Student interests  

The Quantitative Methods (QM) field of study is uniquely designed for mid-career health professionals, research scientists, and MD/MPH students. Students who choose QM are passionate about clinical and population-based health research, and dedicated to learning the tools necessary for implementation.    

Career outcomes

Graduates of the Master of Public Health (MPH) 45-credit program with the Quantitative Methods (QM) field of study are prepared to fulfill professional positions in clinical and population-based health research in government, health care institutions, and private industry.  

Public and patient involvement in quantitative health research: A statistical perspective

Affiliations.

• 1 Public and Patient Involvement Research Unit, Graduate Entry Medical School, University of Limerick, Limerick, Ireland.
• 2 Health Research Institute, University of Limerick, Limerick, Ireland.
• PMID: 29920877
• PMCID: PMC6250860
• DOI: 10.1111/hex.12800

Background: The majority of studies included in recent reviews of impact for public and patient involvement (PPI) in health research had a qualitative design. PPI in solely quantitative designs is underexplored, particularly its impact on statistical analysis. Statisticians in practice have a long history of working in both consultative (indirect) and collaborative (direct) roles in health research, yet their perspective on PPI in quantitative health research has never been explicitly examined.

Objective: To explore the potential and challenges of PPI from a statistical perspective at distinct stages of quantitative research, that is sampling, measurement and statistical analysis, distinguishing between indirect and direct PPI.

Conclusions: Statistical analysis is underpinned by having a representative sample, and a collaborative or direct approach to PPI may help achieve that by supporting access to and increasing participation of under-represented groups in the population. Acknowledging and valuing the role of lay knowledge of the context in statistical analysis and in deciding what variables to measure may support collective learning and advance scientific understanding, as evidenced by the use of participatory modelling in other disciplines. A recurring issue for quantitative researchers, which reflects quantitative sampling methods, is the selection and required number of PPI contributors, and this requires further methodological development. Direct approaches to PPI in quantitative health research may potentially increase its impact, but the facilitation and partnership skills required may require further training for all stakeholders, including statisticians.

Keywords: clinical trial; cohort studies; participatory; public and patient involvement; quantitative; statistics.

© 2018 The Authors Health Expectations published by John Wiley & Sons Ltd.

• Data Interpretation, Statistical*
• Health Services Research*
• Patient Participation*
• Research Design*

You are using an outdated browser! Upgrade your browser today or install Google Chrome Frame to better experience this site.

Search form

Click "Menu" to toggle open, click "Menu" again to close

• What Is Public Health?
• Why Accreditation Matters
• Academic Departments
• Administrative Units
• Office of Information Technology
• Financial Affairs and Physical Resources
• Community, Environment & Policy
• Epidemiology and Biostatistics
• Health Promotion Sciences
• Primary Faculty
• Adjunct Faculty
• Strategic Initiatives
• Centers and Institutes
• Diversity and Inclusion
• Dean's Message
• Dean's Strategic Initiatives Fund
• Dean's Students Research Funds
• Standing Committees
• Need a Public Health Intern?
• Declaring the Public Health Major
• How to Apply for Advanced Standing
• Wellness and Health Promotion Practice (BA)
• Public Health Major (BS)
• Online Bachelors (BS)
• Addiction and Substance Use
• Aging and Population Health Minor
• Climate Change and Public Health
• Environmental and Occupational Health Minor
• Global Health Minor
• One Health Minor
• Population Health Data Science Minor
• Public Health Minor
• Wellness and Health Promotion Practice Minor
• Transfer Students
• Global Health
• Public Health Emergency and Epidemic Preparedness
• Safety Certificate
• Online Degree Curriculum
• Curriculum Information for Public Health Coursework
• Registration Tips for Undergraduates
• Internships
• Study Abroad
• Course Schedule
• BS & MPH Environmental & Occupational Health Program
• BS & MPH in Global Health Program
• Academic Policies and Forms
• Student Services Staff
• Student Clubs and Organizations
• Public Health Ambassadors
• Meet Our Students
• Scholarships and Aid
• Academic Achievement
• Dual Degrees
• MS & PhD Programs
• Certificate Programs
• Non-Degree Seeking
• Distance Learning
• Ajman UAE Micro-Campus
• Study Abroad Programs
• Biostatistics
• Environmental Health Sciences
• Epidemiology
• Family and Child Health
• Health Behavior Health Promotion
• Health Services Administration (Phoenix & Tucson)
• Public Health Policy and Management
• Public Health Practice (Phoenix)
• Prospective Students
• Admissions Criteria & Deadlines
• How to Apply
• Admissions Information Events
• Visiting Campus
• International Students
• Tuition & Financial Assistance
• Certificate Program Admissions Criteria
• Online Team Staff
• Admitted Students
• Research Areas
• Research Projects
• Research Resources
• Research Policies
• Deans Annual Fund Application
• Student Research
• Health Services Administration
• Phoenix Campus
• Public Health Policy & Management
• Public Health Practice
• Center for Firefighter Health Collaborative Research
• Center for Health Disparities Research
• Nosotros Program
• One Health Initiative
• Tucson and Pima County
• Arizona and Regional
• National and Multi-State
• International
• Community Engagement and Outreach
• IndigiWellbeing Program
• Phoenix Mobile Health Unit
• Tucson Mobile Health Unit
• Mobile Outreach Vaccination & Education (MOVE-UP)
• Ventanillas de Salud (Windows to Health)
• Rural Health Professions Program
• Street Medicine Phoenix
• Students Reach Out
• Workforce Development
• 20th Anniversary Stories
• Remembering Mel Zuckerman
• Mental Health
• Climate Change and Health
• Indigenous Health
• Digital Epidemiology
• Additional Giving Opportunities
• Ways to Give
• Scholarships
• Community Advisory Board
• Alumni Giving Circle
• Dean's Circle of Excellence
• Dalen Lecture Series
• Mel & Enid Zuckerman
• James & Priscilla Dalen
• Frank Marcus
• Barry & Janet Lang
• Kent & Liz Campbell
• Meet the Giving Team
• Dean's Annual Fund Application
• Alumni Council
• Host an Event
• Alumni Network
• Alumni Events
• Our Alumni Change the World
• Delta Omega

Quantitative Methods in Public Health Track

Bachelor of science in public health.

Data is the backbone of all public health programming, evaluation, and surveillance. Join the quantitative track and gain skills to answer public health’s most important data-driven questions!

The public health field always has a need for analytical minds for tracking and prevention of disease. The University of Arizona's Bachelor of Science with a major in Public Health is accredited by the Council on Education for Public Health. Students pursuing the Quantitative Methods in Public Health emphasis take courses on the acquisition, assessment, analysis, management and communication of health data. They also study health research and epidemiology. In addition, undergraduates complement their coursework with an internship that provides them with professional public health experience in their area of interest.

LEARNING OUTCOMES

• Students will be able to identify key sources of health data
• Students will be able to implement data management and quality control techniques for a commonly used data entry system
• Students will be able to execute dataset integration and conduct basic statistical analyses on existing datasets
• Students will be able to communicate public health information in written form and using a variety of media
• Students will be able to synthesize epidemiological information

SAMPLE COURSES

• EPID 411: Health and Disease Across Time and the World
• EPID 450: Health Data Acquisition, Assessment, and Integration
• EPID 479: Infections and Epidemics
• BIOS 451: Health Data Management and Visualization
• BIOS 452: Health Data Analysis and Communication Methods
• EPID 453: Health Data Science Practice

CAREER FIELDS

• Public health research
• Data analysis
• Health data management

LEVEL OF MATH

Moderate Knowledge

LEVEL OF SECOND LANGUAGE

2nd Semester Proficiency

PRE-MAJOR REQUIRED?

ADDITIONAL ADMISSIONS REQUIREMENTS

This major has additional admission criteria. Please see here for more information .

For more information about declaring this track for your degree, contact an undergraduate advisor.

Contact Undergraduate Advisor   

BS in Public Health

• Health Promotion
• Environmental and Occupational Health
• Quantitative Methods in Public Health
• Health Systems Theory and Practice
• BS in Public Health Homepage

BA in Wellness and Health Promotion Practice

• Health Education
• Health and Wellness
• Aging and Population Health
• BA in Wellness and Health Promotion Practice Homepage

Explore Available Public Health Minors

Public Health Minors

Online Programs

Certificates

Academic Curriculum Guide

Tucson - Main Campus  |   1295 N. Martin Ave.  |   Drachman Hall  |   P.O. Box 245163  |  Tucson, Arizona 85724 | Fax Number:  520-626-8009 Phoenix Campus  |   550 E. Van Buren Street  |   UA Phoenix Plaza Building 1  |   Phoenix, AZ 85006 The University of Arizona  |   Tucson, Arizona USA 85721-0036  |   Copyright 2024 © Arizona Board of Regents University Privacy Statement

• Systematic review
• Open access
• Published: 19 June 2020

Quantitative measures of health policy implementation determinants and outcomes: a systematic review

• Peg Allen   ORCID: orcid.org/0000-0001-7000-796X 1 ,
• Meagan Pilar 1 ,
• Callie Walsh-Bailey 1 ,
• Cole Hooley 2 ,
• Stephanie Mazzucca 1 ,
• Cara C. Lewis 3 ,
• Kayne D. Mettert 3 ,
• Caitlin N. Dorsey 3 ,
• Jonathan Purtle 4 ,
• Maura M. Kepper 1 ,
• Ana A. Baumann 5 &
• Ross C. Brownson 1 , 6  

Implementation Science volume  15 , Article number:  47 ( 2020 ) Cite this article

18k Accesses

61 Citations

32 Altmetric

Metrics details

Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized. Implementation outcomes include acceptability, adoption, appropriateness, compliance/fidelity, feasibility, penetration, sustainability, and costs. This systematic review identified quantitative measures that are used to assess health policy implementation determinants and outcomes and evaluated the quality of these measures.

Three frameworks guided the review: Implementation Outcomes Framework (Proctor et al.), Consolidated Framework for Implementation Research (Damschroder et al.), and Policy Implementation Determinants Framework (Bullock et al.). Six databases were searched: Medline, CINAHL Plus, PsycInfo, PAIS, ERIC, and Worldwide Political. Searches were limited to English language, peer-reviewed journal articles published January 1995 to April 2019. Search terms addressed four levels: health, public policy, implementation, and measurement. Empirical studies of public policies addressing physical or behavioral health with quantitative self-report or archival measures of policy implementation with at least two items assessing implementation outcomes or determinants were included. Consensus scoring of the Psychometric and Pragmatic Evidence Rating Scale assessed the quality of measures.

Database searches yielded 8417 non-duplicate studies, with 870 (10.3%) undergoing full-text screening, yielding 66 studies. From the included studies, 70 unique measures were identified to quantitatively assess implementation outcomes and/or determinants. Acceptability, feasibility, appropriateness, and compliance were the most commonly measured implementation outcomes. Common determinants in the identified measures were organizational culture, implementation climate, and readiness for implementation, each aspects of the internal setting. Pragmatic quality ranged from adequate to good, with most measures freely available, brief, and at high school reading level. Few psychometric properties were reported.

Conclusions

Well-tested quantitative measures of implementation internal settings were under-utilized in policy studies. Further development and testing of external context measures are warranted. This review is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and determinants to help practitioners and researchers spread evidence-informed policies to improve population health.

Registration

Not registered

Peer Review reports

Contributions to the literature

This systematic review identified 70 quantitative measures of implementation outcomes or determinants in health policy studies.

Readiness to implement and organizational climate and culture were commonly assessed determinants, but fewer studies assessed policy actor relationships or implementation outcomes of acceptability, fidelity/compliance, appropriateness, feasibility, or implementation costs.

Study team members rated most identified measures’ pragmatic properties as good, meaning they are straightforward to use, but few studies documented pilot or psychometric testing of measures.

Further development and dissemination of valid and reliable measures of policy implementation outcomes and determinants can facilitate identification, use, and spread of effective policy implementation strategies.

Despite major impacts of policy on population health [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ], there have been relatively few policy studies in dissemination and implementation (D&I) science to inform implementation strategies and evaluate implementation efforts [ 8 ]. While health outcomes of policies are commonly studied, fewer policy studies assess implementation processes and outcomes. Of 146 D&I studies funded by the National Institutes of Health (NIH) through D&I funding announcements from 2007 to 2014, 12 (8.2%) were policy studies that assessed policy content, policy development processes, or health outcomes of policies, representing 10.5% of NIH D&I funding [ 8 ]. Eight of the 12 studies (66.7%) assessed health outcomes, while only five (41.6%) assessed implementation [ 8 ].

Our ability to explore the differential impact of policy implementation determinants and outcomes and disentangle these from health benefits and other societal outcomes requires high quality quantitative measures [ 9 ]. While systematic reviews of measures of implementation of evidence-based interventions (in clinical and community settings) have been conducted in recent years [ 10 , 11 , 12 , 13 ], to our knowledge, no reviews have explored the quality of quantitative measures of determinants and outcomes of policy implementation.

Policy implementation research in political science and the social sciences has been active since at least the 1970s and has much to contribute to the newer field of D&I research [ 1 , 14 ]. Historically, theoretical frameworks and policy research largely emphasized policy development or analysis of the content of policy documents themselves [ 15 ]. For example, Kingdon’s Multiple Streams Framework and its expansions have been widely used in political science and the social sciences more broadly to describe how factors related to sociopolitical climate, attributes of a proposed policy, and policy actors (e.g., organizations, sectors, individuals) contribute to policy change [ 16 , 17 , 18 ]. Policy frameworks can also inform implementation planning and evaluation in D&I research. Although authors have named policy stages since the 1950s [ 19 , 20 ], Sabatier and Mazmanian’s Policy Implementation Process Framework was one of the first such frameworks that gained widespread use in policy implementation research [ 21 ] and later in health promotion [ 22 ]. Yet, available implementation frameworks are not often used to guide implementation strategies or inform why a policy worked in one setting but not another [ 23 ]. Without explicit focus on implementation, the intended benefits of health policies may go unrealized, and the ability may be lost to move the field forward to understand policy implementation (i.e., our collective knowledge building is dampened) [ 24 ].

Differences in perspectives and terminology between D&I and policy research in political science are noteworthy to interpret the present review. For example, Proctor et al. use the term implementation outcomes for what policy researchers call policy outputs [ 14 , 20 , 25 ]. To non-D&I policy researchers, policy implementation outcomes refer to the health outcomes in the target population [ 20 ]. D&I science uses the term fidelity [ 26 ]; policy researchers write about compliance [ 20 ]. While D&I science uses the terms outer setting, outer context, or external context to point to influences outside the implementing organization [ 26 , 27 , 28 ], non-D&I policy research refers to policy fields [ 24 ] which are networks of agencies that carry out policies and programs.

Identification of valid and reliable quantitative measures of health policy implementation processes is needed. These measures are needed to advance from classifying constructs to understanding causality in policy implementation research [ 29 ]. Given limited resources, policy implementers also need to know which aspects of implementation are key to improve policy acceptance, compliance, and sustainability to reap the intended health benefits [ 30 ]. Both pragmatic and psychometrically sound measures are needed to accomplish these objectives [ 10 , 11 , 31 , 32 ], so the field can explore the influence of nuanced determinants and generate reliable and valid findings.

To fill this void in the literature, this systematic review of health policy implementation measures aimed to (1) identify quantitative measures used to assess health policy implementation outcomes (IOF outcomes commonly called policy outputs in policy research) and inner and outer setting determinants, (2) describe and assess pragmatic quality of policy implementation measures, (3) describe and assess the quality of psychometric properties of identified instruments, and (4) elucidate health policy implementation measurement gaps.

The study team used systematic review procedures developed by Lewis and colleagues for reviews of D&I research measures and received detailed guidance from the Lewis team coauthors for each step [ 10 , 11 ]. We followed the PRISMA reporting guidelines as shown in the checklist (Supplemental Table 1 ). We have also provided a publicly available website of measures identified in this review ( https://www.health-policy-measures.org/ ).

For the purposes of this review, policy and policy implementation are defined as follows. We deemed public policy to include legislation at the federal, state/province/regional unit, or local levels; and governmental regulations, whether mandated by national, state/province, or local level governmental agencies or boards of elected officials (e.g., state boards of education in the USA) [ 4 , 20 ]. Here, public policy implementation is defined as the carrying out of a governmental mandate by public or private organizations and groups of organizations [ 20 ].

Two widely used frameworks from the D&I field guide the present review, and a third recently developed framework that bridges policy and D&I research. In the Implementation Outcomes Framework (IOF), Proctor and colleagues identify and define eight implementation outcomes that are differentiated from health outcomes: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability [ 25 ]. In the Consolidated Framework for Implementation Research (CFIR), Damschroder and colleagues articulate determinants of implementation including the domains of intervention characteristics, outer setting, inner setting of an organization, characteristics of individuals within organizations, and process [ 33 ]. Finally, Bullock developed the Policy Implementation Determinants Framework to present a balanced framework that emphasizes both internal setting constructs and external setting constructs including policy actor relationships and networks, political will for implementation, and visibility of policy actors [ 34 ]. The constructs identified in these frameworks were used to guide our list of implementation determinants and outcomes.

Through EBSCO, we searched MEDLINE, PsycInfo, and CINAHL Plus. Through ProQuest, we searched PAIS, Worldwide Political, and ERIC. Due to limited time and staff in the 12-month study, we did not search the grey literature. We used multiple search terms in each of four required levels: health, public policy, implementation, and measurement (Table 1 ). Table 1 shows search terms for each string. Supplemental Tables 2 and 3 show the final search syntax applied in EBSCO and ProQuest.

The authors developed the search strings and terms based on policy implementation framework reviews [ 34 , 35 ], additional policy implementation frameworks [ 21 , 22 ], labels and definitions of the eight implementation outcomes identified by Proctor et al. [ 25 ], CFIR construct labels and definitions [ 9 , 33 ], and additional D&I research and search term sources [ 28 , 36 , 37 , 38 ] (Table 1 ). The full study team provided three rounds of feedback on draft terms, and a library scientist provided additional synonyms and search terms. For each test search, we calculated the percentage of 18 benchmark articles the search captured. We determined a priori 80% as an acceptable level of precision.

Inclusion and exclusion criteria

This review addressed only measures of implementation by organizations mandated to act by governmental units or legislation. Measures of behavior changes by individuals in target populations as a result of legislation or governmental regulations and health status changes were outside the realm of this review.

There were several inclusion criteria: (1) empirical studies of the implementation of public policies already passed or approved that addressed physical or behavioral health, (2) quantitative self-report or archival measurement methods utilized, (3) published in peer-reviewed journals from January 1995 through April 2019, (4) published in the English language, (5) public policy implementation studies from any continent or international governing body, and (6) at least two transferable quantitative self-report or archival items that assessed implementation determinants [ 33 , 34 ] and/or IOF implementation outcomes [ 25 ]. This study sought to identify transferable measures that could be used to assess multiple policies and contexts. Here, a transferable item is defined as one that needed no wording changes or only a change in the referent (e.g., policy title or topic such as tobacco or malaria) to make the item applicable to other policies or settings [ 11 ]. The year 1995 was chosen as a starting year because that is about when web-based quantitative surveying began [ 39 ]. Table 2 provides definitions of the IOF implementation outcomes and the selected determinants of implementation. Broader constructs, such as readiness for implementation, contained multiple categories.

Exclusion criteria in the searches included (1) non-empiric health policy journal articles (e.g., conceptual articles, editorials); (2) narrative and systematic reviews; (3) studies with only qualitative assessment of health policy implementation; (4) empiric studies reported in theses and books; (5) health policy studies that only assessed health outcomes (i.e., target population changes in health behavior or status); (6) bill analyses, stakeholder perceptions assessed to inform policy development, and policy content analyses without implementation assessment; (7) studies of changes made in a private business not encouraged by public policy; and (8) countries with authoritarian regimes. We electronically programmed the searches to exclude policy implementation studies from countries that are not democratically governed due to vast differences in policy environments and implementation factors.

Screening procedures

Citations were downloaded into EndNote version 7.8 and de-duplicated electronically. We conducted dual independent screening of titles and abstracts after two group pilot screening sessions in which we clarified inclusion and exclusion criteria and screening procedures. Abstract screeners used Covidence systematic review software [ 40 ] to code inclusion as yes or no. Articles were included in full-text review if one screener coded it as meeting the inclusion criteria. Full-text screening via dual independent screening was coded in Covidence [ 40 ], with weekly meetings to reach consensus on inclusion/exclusion discrepancies. Screeners also coded one of the pre-identified reasons for exclusion.

Data extraction strategy

Extraction elements included information about (1) measure meta-data (e.g., measure name, total number of items, number of transferable items) and studies (e.g., policy topic, country, setting), (2) development and testing of the measure, (3) implementation outcomes and determinants assessed (Table 2 ), (4) pragmatic characteristics, and (5) psychometric properties. Where needed, authors were emailed to obtain the full measure and measure development information. Two coauthors (MP, CWB) reached consensus on extraction elements. For each included measure, a primary extractor conducted initial entries and coding. Due to time and staff limitations in the 12-month study, we did not search for each empirical use of the measure. A secondary extractor checked the entries, noting any discrepancies for discussion in consensus meetings. Multiple measures in a study were extracted separately.

Quality assessment of measures

To assess the quality of measures, we applied the Psychometric and Pragmatic Evidence Rating Scales (PAPERS) developed by Lewis et al. [ 10 , 11 , 41 , 42 ]. PAPERS includes assessment of five pragmatic instrument characteristics that affect the level of ease or difficulty to use the instrument: brevity (number of items), simplicity of language (readability level), cost (whether it is freely available), training burden (extent of data collection training needed), and analysis burden (ease or difficulty of interpretation of scoring and results). Lewis and colleagues developed the pragmatic domains and rating scales with stakeholder and D&I researchers input [ 11 , 41 , 42 ] and developed the psychometric rating scales in collaboration with D&I researchers [ 10 , 11 , 43 ]. The psychometric rating scale has nine properties (Table 3 ): internal consistency; norms; responsiveness; convergent, discriminant, and known-groups construct validity; predictive and concurrent criterion validity; and structural validity. In both the pragmatic and psychometric scales, reported evidence for each domain is scored from poor (− 1), none/not reported (0), minimal/emerging (1), adequate (2), good (3), or excellent (4). Higher values are indicative of more desirable pragmatic characteristics (e.g., fewer items, freely available, scoring instructions, and interpretations provided) and stronger evidence of psychometric properties (e.g., adequate to excellent reliability and validity) (Supplemental Tables 4 and 5 ).

Data synthesis and presentation

This section describes the synthesis of measure transferability, empiric use study settings and policy topics, and PAPERS scoring. Two coauthors (MP, CWB) consensus coded measures into three categories of item transferability based on quartile item transferability percentages: mostly transferable (≥ 75% of items deemed transferable), partially transferable (25–74% of items deemed transferable), and setting-specific (< 25% of items deemed transferable). Items were deemed transferable if no wording changes or only a change in the referent (e.g., policy title or topic) was needed to make the item applicable to the implementation of other policies or in other settings. Abstractors coded study settings into one of five categories: hospital or outpatient clinics; mental or behavioral health facilities; healthcare cost, access, or quality; schools; community; and multiple. Abstractors also coded policy topics to healthcare cost, access, or quality; mental or behavioral health; infectious or chronic diseases; and other, while retaining documentation of subtopics such as tobacco, physical activity, and nutrition. Pragmatic scores were totaled for the five properties, with possible total scores of − 5 to 20, with higher values indicating greater ease to use the instrument. Psychometric property total scores for the nine properties were also calculated, with possible scores of − 9 to 36, with higher values indicating evidence of multiple types of validity.

The database searches yielded 11,684 articles, of which 3267 were duplicates (Fig. 1 ). Titles and abstracts of the 8417 articles were independently screened by two team members; 870 (10.3%) were selected for full-text screening by at least one screener. Of the 870 studies, 804 were excluded at full-text screening or during extraction attempts with the consensus of two coauthors; 66 studies were included. Two coauthors (MP, CWB) reached consensus on extraction and coding of information on 70 unique quantitative eligible measures identified in the 66 included studies plus measure development articles where obtained. Nine measures were used in more than one included study. Detailed information on identified measures is publicly available at https://www.health-policy-measures.org/ .

PRISMA flow diagram

The most common exclusion reason was lack of transferable items in quantitative measures of policy implementation ( n = 597) (Fig. 1 ). While this review focused on transferable measures across any health issue or setting, researchers addressing specific health policies or settings may find the excluded studies of interest. The frequencies of the remaining exclusion reasons are listed in Fig. 1 .

A variety of health policy topics and settings from over two dozen countries were found in the database searches. For example, the searches identified quantitative and mixed methods implementation studies of legislation (such as tobacco smoking bans), regulations (such as food/menu labeling requirements), governmental policies that mandated specific clinical practices (such as vaccination or access to HIV antiretroviral treatment), school-based interventions (such as government-mandated nutritional content and physical activity), and other public policies.

Among the 70 unique quantitative implementation measures, 15 measures were deemed mostly transferable (at least 75% transferable, Table 4 ). Twenty-three measures were categorized as partially transferable (25 to 74% of items deemed transferable, Table 5 ); 32 measures were setting-specific (< 25% of items deemed transferable, data not shown).

Implementation outcomes

Among the 70 measures, the most commonly assessed implementation outcomes were fidelity/compliance of the policy implementation to the government mandate (26%), acceptability of the policy to implementers (24%), perceived appropriateness of the policy (17%), and feasibility of implementation (17%) (Table 2 ). Fidelity/compliance was sometimes assessed by asking implementers the extent to which they had modified a mandated practice [ 45 ]. Sometimes, detailed checklists were used to assess the extent of compliance with the many mandated policy components, such as school nutrition policies [ 83 ]. Acceptability was assessed by asking staff or healthcare providers in implementing agencies their level of agreement with the provided statements about the policy mandate, scored in Likert scales. Only eight (11%) of the included measures used multiple transferable items to assess adoption, and only eight (11%) assessed penetration.

Twenty-six measures of implementation costs were found during full-text screening (10 in included studies and 14 in excluded studies, data not shown). The cost time horizon varied from 12 months to 21 years, with most cost measures assessed at multiple time points. Ten of the 26 measures addressed direct implementation costs. Nine studies reported cost modeling findings. The implementation cost survey developed by Vogler et al. was extensive [ 53 ]. It asked implementing organizations to note policy impacts in medication pricing, margins, reimbursement rates, and insurance co-pays.

Determinants of implementation

Within the 70 included measures, the most commonly assessed implementation determinants were readiness for implementation (61% assessed any readiness component) and the general organizational culture and climate (39%), followed by the specific policy implementation climate within the implementation organization/s (23%), actor relationships and networks (17%), political will for policy implementation (11%), and visibility of the policy role and policy actors (10%) (Table 2 ). Each component of readiness for implementation was commonly assessed: communication of the policy (31%, 22 of 70 measures), policy awareness and knowledge (26%), resources for policy implementation (non-training resources 27%, training 20%), and leadership commitment to implement the policy (19%).

Only two studies assessed organizational structure as a determinant of health policy implementation. Lavinghouze and colleagues assessed the stability of the organization, defined as whether re-organization happens often or not, within a set of 9-point Likert items on multiple implementation determinants designed for use with state-level public health practitioners, and assessed whether public health departments were stand-alone agencies or embedded within agencies addressing additional services, such as social services [ 69 ]. Schneider and colleagues assessed coalition structure as an implementation determinant, including items on the number of organizations and individuals on the coalition roster, number that regularly attend coalition meetings, and so forth [ 72 ].

Tables of measures

Tables 4 and 5 present the 38 measures of implementation outcomes and/or determinants identified out of the 70 included measures with at least 25% of items transferable (useable in other studies without wording changes or by changing only the policy name or other referent). Table 4 shows 15 mostly transferable measures (at least 75% transferable). Table 5 shows 23 partially transferable measures (25–74% of items deemed transferable). Separate measure development articles were found for 20 of the 38 measures; the remaining measures seemed to be developed for one-time, study-specific use by the empirical study authors cited in the tables. Studies listed in Tables 4 and 5 were conducted most commonly in the USA ( n = 19) or Europe ( n = 11). A few measures were used elsewhere: Africa ( n = 3), Australia ( n = 1), Canada ( n = 1), Middle East ( n = 1), Southeast Asia ( n = 1), or across multiple continents ( n = 1).

Quality of identified measures

Figure 2 shows the median pragmatic quality ratings across the 38 measures with at least 25% transferable items shown in Tables 4 and 5 . Higher scores are desirable and indicate the measures are easier to use (Table 3 ). Overall, the measures were freely available in the public domain (median score = 4), brief with a median of 11–50 items (median score = 3), and had good readability, with a median reading level between 8th and 12th grade (median score = 3). However, instructions on how to score and interpret item scores were lacking, with a median score of 1, indicating the measures did not include suggestions for interpreting score ranges, clear cutoff scores, and instructions for handling missing data. In general, information on training requirements or availability of self-training manuals on how to use the measures was not reported in the included study or measure development article/s (median score = 0, not reported). Total pragmatic rating scores among the 38 measures with at least 25% of items transferable ranged from 7 to 17 (Tables 4 and 5 ), with a median total score of 12 out of a possible total score of 20. Median scores for each pragmatic characteristic were the same across all measures as for the 38 mostly or partially transferable measures, with a median total score of 11 across all measures.

Pragmatic rating scale results across identified measures. Footnote: pragmatic criteria scores from Psychometric and Pragmatic Evidence Rating Scale (PAPERS) (Lewis et al. [ 11 ], Stanick et al. [ 42 ]). Total possible score = 20, total median score across 38 measures = 11. Scores ranged from 0 to 18. Rating scales for each domain are provided in Supplemental Table 4

Few psychometric properties were reported. The study team found few reports of pilot testing and measure refinement as well. Among the 38 measures with at least 25% transferable items, the psychometric properties from the PAPERS rating scale total scores ranged from − 1 to 17 (Tables 4 and 5 ), with a median total score of 5 out of a possible total score of 36. Higher scores indicate more types of validity and reliability were reported with high quality. The 32 measures with calculable norms had a median norms PAPERS score of 3 (good), indicating appropriate sample size and distribution. The nine measures with reported internal consistency mostly showed Cronbach’s alphas in the adequate (0.70 to 0.79) to excellent (≥ 90) range, with a median of 0.78 (PAPERS score of 2, adequate) indicating adequate internal consistency. The five measures with reported structural validity had a median PAPERS score of 2, adequate (range 1 to 3, poor to good), indicating the sample size was sufficient and the factor analysis goodness of fit was reasonable. Among the 38 measures, no reports were found for responsiveness, convergent validity, discriminant validity, known-groups construct validity, or predictive or concurrent criterion validity.

In this systematic review, we sought to identify quantitative measures used to assess health policy implementation outcomes and determinants, rate the pragmatic and psychometric quality of identified measures, and point to future directions to address measurement gaps. In general, the identified measures are easy to use and freely available, but we found little data on validity and reliability. We found more quantitative measures of intra-organizational determinants of policy implementation than measures of the relationships and interactions between organizations that influence policy implementation. We found a limited number of measures that had been developed for or used to assess one of the eight IOF policy implementation outcomes that can be applied to other policies or settings, which may speak more to differences in terms used by policy researchers and D&I researchers than to differences in conceptualizations of policy implementation. Authors used a variety of terms and rarely provided definitions of the constructs the items assessed. Input from experts in policy implementation is needed to better understand and define policy implementation constructs for use across multiple fields involved in policy-related research.

We found several researchers had used well-tested measures of implementation determinants from D&I research or from organizational behavior and management literature (Tables 4 and 5 ). For internal setting of implementing organizations, whether mandated through public policy or not, well-developed and tested measures are available. However, a number of authors crafted their own items, with or without pilot testing, and used a variety of terms to describe what the items assessed. Further dissemination of the availability of well-tested measures to policy researchers is warranted [ 9 , 13 ].

What appears to be a larger gap involves the availability of well-developed and tested quantitative measures of the external context affecting policy implementation that can be used across multiple policy settings and topics [ 9 ]. Lack of attention to how a policy initiative fits with the external implementation context during policymaking and lack of policymaker commitment of adequate resources for implementation contribute to this gap [ 23 , 93 ]. Recent calls and initiatives to integrate health policies during policymaking and implementation planning will bring more attention to external contexts affecting not only policy development but implementation as well [ 93 , 94 , 95 , 96 , 97 , 98 , 99 ]. At the present time, it is not well-known which internal and external determinants are most essential to guide and achieve sustainable policy implementation [ 100 ]. Identification and dissemination of measures that assess factors that facilitate the spread of evidence-informed policy implementation (e.g., relative advantage, flexibility) will also help move policy implementation research forward [ 1 , 9 ].

Given the high potential population health impact of evidence-informed policies, much more attention to policy implementation is needed in D&I research. Few studies from non-D&I researchers reported policy implementation measure development procedures, pilot testing, scoring procedures and interpretation, training of data collectors, or data analysis procedures. Policy implementation research could benefit from the rigor of D&I quantitative research methods. And D&I researchers have much to learn about the contexts and practical aspects of policy implementation and can look to the rich depth of information in qualitative and mixed methods studies from other fields to inform quantitative measure development and testing [ 101 , 102 , 103 ].

Limitations

This systematic review has several limitations. First, the four levels of the search string and multiple search terms in each level were applied only to the title, abstract, and subject headings, due to limitations of the search engines, so we likely missed pertinent studies. Second, a systematic approach with stakeholder input is needed to expand the definitions of IOF implementation outcomes for policy implementation. Third, although the authors value intra-organizational policymaking and implementation, the study team restricted the search to governmental policies due to limited time and staffing in the 12-month study. Fourth, by excluding tools with only policy-specific implementation measures, we excluded some well-developed and tested instruments in abstract and full-text screening. Since only 12 measures had 100% transferable items, researchers may need to pilot test wording modifications of other items. And finally, due to limited time and staffing, we only searched online for measures and measures development articles and may have missed separately developed pragmatic information, such as training and scoring materials not reported in a manuscript.

Despite the limitations, several recommendations for measure development follow from the findings and related literature [ 1 , 11 , 20 , 35 , 41 , 104 ], including the need to (1) conduct systematic, mixed-methods procedures (concept mapping, expert panels) to refine policy implementation outcomes, (2) expand and more fully specify external context domains for policy implementation research and evaluation, (3) identify and disseminate well-developed measures for specific policy topics and settings, (4) ensure that policy implementation improves equity rather than exacerbating disparities [ 105 ], and (5) develop evidence-informed policy implementation guidelines.

Easy-to-use, reliable, and valid quantitative measures of policy implementation can further our understanding of policy implementation processes, determinants, and outcomes. Due to the wide array of health policy topics and implementation settings, sound quantitative measures that can be applied across topics and settings will help speed learnings from individual studies and aid in the transfer from research to practice. Quantitative measures can inform the implementation of evidence-informed policies to further the spread and effective implementation of policies to ultimately reap greater population health benefit. This systematic review of measures is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and predictors to help practitioners and researchers spread evidence-informed policies to improve population health and reduce inequities.

Availability of data and materials

A compendium of identified measures is available for dissemination at https://www.health-policy-measures.org/ . A link will be provided on the website of the Prevention Research Center, Brown School, Washington University in St. Louis, at https://prcstl.wustl.edu/ . The authors invite interested organizations to provide a link to the compendium. Citations and abstracts of excluded policy-specific measures are available on request.

Abbreviations

Consolidated Framework for Implementation Research

Cumulative Index of Nursing and Allied Health Literature

Dissemination and implementation science

Elton B. Stephens Company

Education Resources Information Center

Implementation Outcomes Framework

Psychometric and Pragmatic Evidence Rating Scale

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Purtle J, Dodson EA, Brownson RC. Policy dissemination research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice, Second Edition. New York: Oxford University Press; 2018.

Google Scholar  

Brownson RC, Baker EA, Deshpande AD, Gillespie KN. Evidence-based public health. Third ed. New York, NY: Oxford University Press; 2018.

Guide to Community Preventive Services. About the community guide.: community preventive services task force; 2020 [updated October 03, 2019; cited 2020. Available from: https://www.thecommunityguide.org/ .

Eyler AA, Chriqui JF, Moreland-Russell S, Brownson RC, editors. Prevention, policy, and public health, first edition. New York, NY: Oxford University Press; 2016.

Andre FE, Booy R, Bock HL, Clemens J, Datta SK, John TJ, et al. Vaccination greatly reduces disease, disability, death, and inequity worldwide. Geneva, Switzerland: World Health Organization; 2008 February 2008. Contract No.: 07-040089.

Cheng JJ, Schuster-Wallace CJ, Watt S, Newbold BK, Mente A. An ecological quantification of the relationships between water, sanitation and infant, child, and maternal mortality. Environ Health. 2012;11:4.

PubMed   PubMed Central   Google Scholar  

Levy DT, Li Y, Yuan Z. Impact of nations meeting the MPOWER targets between 2014 and 2016: an update. Tob Control. 2019.

Purtle J, Peters R, Brownson RC. A review of policy dissemination and implementation research funded by the National Institutes of Health, 2007-2014. Implement Sci. 2016;11:1.

Lewis CC, Proctor EK, Brownson RC. Measurement issues in dissemination and implementation research. In: Brownson RC, Ga C, Proctor EK, editors. Disssemination and Implementation Research in Health: Translating Science to Practice, Second Edition. New York: Oxford University Press; 2018.

Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10:155.

Lewis CC, Mettert KD, Dorsey CN, Martinez RG, Weiner BJ, Nolen E, et al. An updated protocol for a systematic review of implementation-related measures. Syst Rev. 2018;7(1):66.

Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.

Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN, et al. Measurement resources for dissemination and implementation research in health. Implement Sci. 2016;11:42.

Nilsen P, Stahl C, Roback K, Cairney P. Never the twain shall meet?--a comparison of implementation science and policy implementation research. Implement Sci. 2013;8:63.

Sabatier PA, editor. Theories of the Policy Process. New York, NY: Routledge; 2019.

Kingdon J. Agendas, alternatives, and public policies, second edition. Second ed. New York: Longman; 1995.

Jones MD, Peterson HL, Pierce JJ, Herweg N, Bernal A, Lamberta Raney H, et al. A river runs through it: a multiple streams meta-review. Policy Stud J. 2016;44(1):13–36.

Fowler L. Using the multiple streams framework to connect policy adoption to implementation. Policy Studies Journal. 2020 (11 Feb).

Howlett M, Mukherjee I, Woo JJ. From tools to toolkits in policy design studies: the new design orientation towards policy formulation research. Policy Polit. 2015;43(2):291–311.

Natesan SD, Marathe RR. Literature review of public policy implementation. Int J Public Policy. 2015;11(4):219–38.

Sabatier PA, Mazmanian. Implementation of public policy: a framework of analysis. Policy Studies Journal. 1980 (January).

Sabatier PA. Theories of the Policy Process. Westview; 2007.

Tomm-Bonde L, Schreiber RS, Allan DE, MacDonald M, Pauly B, Hancock T, et al. Fading vision: knowledge translation in the implementation of a public health policy intervention. Implement Sci. 2013;8:59.

Roll S, Moulton S, Sandfort J. A comparative analysis of two streams of implementation research. Journal of Public and Nonprofit Affairs. 2017;3(1):3–22.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38(2):65–76.

Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice, second edition. New York: Oxford University Press; 2018.

Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50.

Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117–23.

PubMed   Google Scholar  

Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:136.

Boyd MR, Powell BJ, Endicott D, Lewis CC. A method for tracking implementation strategies: an exemplar implementing measurement-based care in community behavioral health clinics. Behav Ther. 2018;49(4):525–37.

Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013;40(3):257–65.

Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013;45(2):237–43.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

Bullock HL. Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis in: How do systems achieve their goals? the role of implementation in mental health systems improvement [Dissertation]. Hamilton, Ontario: McMaster University; 2019.

Watson DP, Adams EL, Shue S, Coates H, McGuire A, Chesher J, et al. Defining the external implementation context: an integrative systematic literature review. BMC Health Serv Res. 2018;18(1):209.

McKibbon KA, Lokker C, Wilczynski NL, Ciliska D, Dobbins M, Davis DA, et al. A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel? Implement Sci. 2010;5:16.

Terwee CB, Jansma EP, Riphagen II, de Vet HC. Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res. 2009;18(8):1115–23.

Egan M, Maclean A, Sweeting H, Hunt K. Comparing the effectiveness of using generic and specific search terms in electronic databases to identify health outcomes for a systematic review: a prospective comparative study of literature search method. BMJ Open. 2012;2:3.

Dillman DA, Smyth JD, Christian LM. Internet, mail, and mixed-mode surveys: the tailored design method. Hoboken, NJ: John Wiley & Sons; 2009.

Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation. https://www.covidence.org . Accessed Mar 2019.

Powell BJ, Stanick CF, Halko HM, Dorsey CN, Weiner BJ, Barwick MA, et al. Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implement Sci. 2017;12(1):118.

Stanick CF, Halko HM, Nolen EA, Powell BJ, Dorsey CN, Mettert KD, et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transl Behav Med. 2019.

Henrikson NB, Blasi PR, Dorsey CN, Mettert KD, Nguyen MB, Walsh-Bailey C, et al. Psychometric and pragmatic properties of social risk screening tools: a systematic review. Am J Prev Med. 2019;57(6S1):S13–24.

Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65.

Lau AS, Brookman-Frazee L. The 4KEEPS study: identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implement Sci. 2016;11:1–8.

Ekvall G. Organizational climate for creativity and innovation. European J Work Organizational Psychology. 1996;5(1):105–23.

Lövgren G, Eriksson S, Sandman PO. Effects of an implemented care policy on patient and personnel experiences of care. Scand J Caring Sci. 2002;16(1):3–11.

Dwyer DJ, Ganster DC. The effects of job demands and control on employee attendance and satisfaction. J Organ Behav. 1991;12:595–608.

Condon-Paoloni D, Yeatman HR, Grigonis-Deane E. Health-related claims on food labels in Australia: understanding environmental health officers’ roles and implications for policy. Public Health Nutr. 2015;18(1):81–8.

Patterson MG, West MA, Shackleton VJ, Dawson JF, Lawthom R, Maitlis S, et al. Validating the organizational climate measure: links to managerial practices, productivity and innovation. J Organ Behav. 2005;26:279–408.

Glisson C, Green P, Williams NJ. Assessing the Organizational Social Context (OSC) of child welfare systems: implications for research and practice. Child Abuse Negl. 2012;36(9):621–32.

Beidas RS, Aarons G, Barg F, Evans A, Hadley T, Hoagwood K, et al. Policy to implementation: evidence-based practice in community mental health--study protocol. Implement Sci. 2013;8(1):38.

Eisenberger R, Cummings J, Armeli S, Lynch P. Perceived organizational support, discretionary treatment, and job satisfaction. J Appl Psychol. 1997;82:812–20.

CAS   PubMed   Google Scholar  

Eby L, George K, Brown BL. Going tobacco-free: predictors of clinician reactions and outcomes of the NY state office of alcoholism and substance abuse services tobacco-free regulation. J Subst Abus Treat. 2013;44(3):280–7.

Vogler S, Zimmermann N, de Joncheere K. Policy interventions related to medicines: survey of measures taken in European countries during 2010-2015. Health Policy. 2016;120(12):1363–77.

Wanberg CRB, Banas JT. Predictors and outcomes of openness to change in a reorganizing workplace. J Applied Psychology. 2000;85:132–42.

CAS   Google Scholar  

Hardy LJ, Wertheim P, Bohan K, Quezada JC, Henley E. A model for evaluating the activities of a coalition-based policy action group: the case of Hermosa Vida. Health Promot Pract. 2013;14(4):514–23.

Gavriilidis G, Östergren P-O. Evaluating a traditional medicine policy in South Africa: phase 1 development of a policy assessment tool. Glob Health Action. 2012;5:17271.

Hongoro C, Rutebemberwa E, Twalo T, Mwendera C, Douglas M, Mukuru M, et al. Analysis of selected policies towards universal health coverage in Uganda: the policy implementation barometer protocol. Archives Public Health. 2018;76:12.

Roeseler A, Solomon M, Beatty C, Sipler AM. The tobacco control network’s policy readiness and stage of change assessment: what the results suggest for moving tobacco control efforts forward at the state and territorial levels. J Public Health Manag Pract. 2016;22(1):9–19.

Brämberg EB, Klinga C, Jensen I, Busch H, Bergström G, Brommels M, et al. Implementation of evidence-based rehabilitation for non-specific back pain and common mental health problems: a process evaluation of a nationwide initiative. BMC Health Serv Res. 2015;15(1):79.

Rütten A, Lüschen G, von Lengerke T, Abel T, Kannas L, Rodríguez Diaz JA, et al. Determinants of health policy impact: comparative results of a European policymaker study. Sozial-Und Praventivmedizin. 2003;48(6):379–91.

Smith SN, Lai Z, Almirall D, Goodrich DE, Abraham KM, Nord KM, et al. Implementing effective policy in a national mental health reengagement program for veterans. J Nerv Ment Dis. 2017;205(2):161–70.

Carasso BS, Lagarde M, Cheelo C, Chansa C, Palmer N. Health worker perspectives on user fee removal in Zambia. Hum Resour Health. 2012;10:40.

Goldsmith REH, C.F. Measuring consumer innovativeness. J Acad Mark Sci. 1991;19(3):209–21.

Webster CA, Caputi P, Perreault M, Doan R, Doutis P, Weaver RG. Elementary classroom teachers’ adoption of physical activity promotion in the context of a statewide policy: an innovation diffusion and socio-ecologic perspective. J Teach Phys Educ. 2013;32(4):419–40.

Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, Cafri G. Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychol Assess. 2010;22(2):356–65.

Gill KJ, Campbell E, Gauthier G, Xenocostas S, Charney D, Macaulay AC. From policy to practice: implementing frontline community health services for substance dependence--study protocol. Implement Sci. 2014;9:108.

Lavinghouze SR, Price AW, Parsons B. The environmental assessment instrument: harnessing the environment for programmatic success. Health Promot Pract. 2009;10(2):176–85.

Bull FC, Milton K, Kahlmeier S. National policy on physical activity: the development of a policy audit tool. J Phys Act Health. 2014;11(2):233–40.

Bull F, Milton K, Kahlmeier S, Arlotti A, Juričan AB, Belander O, et al. Turning the tide: national policy approaches to increasing physical activity in seven European countries. British J Sports Med. 2015;49(11):749–56.

Schneider EC, Smith ML, Ory MG, Altpeter M, Beattie BL, Scheirer MA, et al. State fall prevention coalitions as systems change agents: an emphasis on policy. Health Promot Pract. 2016;17(2):244–53.

Helfrich CD, Savitz LA, Swiger KD, Weiner BJ. Adoption and implementation of mandated diabetes registries by community health centers. Am J Prev Med. 2007;33(1,Suppl):S50-S65.

Donchin M, Shemesh AA, Horowitz P, Daoud N. Implementation of the Healthy Cities’ principles and strategies: an evaluation of the Israel Healthy Cities network. Health Promot Int. 2006;21(4):266–73.

Were MC, Emenyonu N, Achieng M, Shen C, Ssali J, Masaba JP, et al. Evaluating a scalable model for implementing electronic health records in resource-limited settings. J Am Med Inform Assoc. 2010;17(3):237–44.

Konduri N, Sawyer K, Nizova N. User experience analysis of e-TB Manager, a nationwide electronic tuberculosis recording and reporting system in Ukraine. ERJ Open Research. 2017;3:2.

McDonnell E, Probart C. School wellness policies: employee participation in the development process and perceptions of the policies. J Child Nutr Manag. 2008;32:1.

Mersini E, Hyska J, Burazeri G. Evaluation of national food and nutrition policy in Albania. Zdravstveno Varstvo. 2017;56(2):115–23.

Cavagnero E, Daelmans B, Gupta N, Scherpbier R, Shankar A. Assessment of the health system and policy environment as a critical complement to tracking intervention coverage for maternal, newborn, and child health. Lancet. 2008;371 North American Edition(9620):1284-93.

Lehman WE, Greener JM, Simpson DD. Assessing organizational readiness for change. J Subst Abus Treat. 2002;22(4):197–209.

Pankratz M, Hallfors D, Cho H. Measuring perceptions of innovation adoption: the diffusion of a federal drug prevention policy. Health Educ Res. 2002;17(3):315–26.

Cook JM, Thompson R, Schnurr PP. Perceived characteristics of intervention scale: development and psychometric properties. Assessment. 2015;22(6):704–14.

Probart C, McDonnell ET, Jomaa L, Fekete V. Lessons from Pennsylvania’s mixed response to federal school wellness law. Health Aff. 2010;29(3):447–53.

Probart C, McDonnell E, Weirich JE, Schilling L, Fekete V. Statewide assessment of local wellness policies in Pennsylvania public school districts. J Am Diet Assoc. 2008;108(9):1497–502.

Rakic S, Novakovic B, Stevic S, Niskanovic J. Introduction of safety and quality standards for private health care providers: a case-study from the Republic of Srpska, Bosnia and Herzegovina. Int J Equity Health. 2018;17(1):92.

Rozema AD, Mathijssen JJP, Jansen MWJ, van Oers JAM. Sustainability of outdoor school ground smoking bans at secondary schools: a mixed-method study. Eur J Pub Health. 2018;28(1):43–9.

Barbero C, Moreland-Russell S, Bach LE, Cyr J. An evaluation of public school district tobacco policies in St. Louis County, Missouri. J Sch Health. 2013;83(8):525–32.

Williams KM, Kirsh S, Aron D, Au D, Helfrich C, Lambert-Kerzner A, et al. Evaluation of the Veterans Health Administration’s specialty care transformational initiatives to promote patient-centered delivery of specialty care: a mixed-methods approach. Telemed J E-Health. 2017;23(7):577–89.

Spencer E, Walshe K. National quality improvement policies and strategies in European healthcare systems. Quality Safety Health Care. 2009;18(Suppl 1):i22–i7.

Assunta M, Dorotheo EU. SEATCA Tobacco Industry Interference Index: a tool for measuring implementation of WHO Framework Convention on Tobacco Control Article 5.3. Tob Control. 2016;25(3):313–8.

Tummers L. Policy alienation of public professionals: the construct and its measurement. Public Adm Rev. 2012;72(4):516–25.

Tummers L, Bekkers V. Policy implementation, street-level bureaucracy, and the importance of discretion. Public Manag Rev. 2014;16(4):527–47.

Raghavan R, Bright CL, Shadoin AL. Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implement Sci. 2008;3:26.

Peters D, Harting J, van Oers H, Schuit J, de Vries N, Stronks K. Manifestations of integrated public health policy in Dutch municipalities. Health Promot Int. 2016;31(2):290–302.

Tosun J, Lang A. Policy integration: mapping the different concepts. Policy Studies. 2017;38(6):553–70.

Tubbing L, Harting J, Stronks K. Unravelling the concept of integrated public health policy: concept mapping with Dutch experts from science, policy, and practice. Health Policy. 2015;119(6):749–59.

Donkin A, Goldblatt P, Allen J, Nathanson V, Marmot M. Global action on the social determinants of health. BMJ Glob Health. 2017;3(Suppl 1):e000603-e.

Baum F, Friel S. Politics, policies and processes: a multidisciplinary and multimethods research programme on policies on the social determinants of health inequity in Australia. BMJ Open. 2017;7(12):e017772-e.

Delany T, Lawless A, Baum F, Popay J, Jones L, McDermott D, et al. Health in All Policies in South Australia: what has supported early implementation? Health Promot Int. 2016;31(4):888–98.

Valaitis R, MacDonald M, Kothari A, O'Mara L, Regan S, Garcia J, et al. Moving towards a new vision: implementation of a public health policy intervention. BMC Public Health. 2016;16:412.

Bennett LM, Gadlin H, Marchand, C. Collaboration team science: a field guide. Bethesda, MD: National Cancer Institute, National Institutes of Health; 2018. Contract No.: NIH Publication No. 18-7660.

Mazumdar M, Messinger S, Finkelstein DM, Goldberg JD, Lindsell CJ, Morton SC, et al. Evaluating academic scientists collaborating in team-based research: a proposed framework. Acad Med. 2015;90(10):1302–8.

Brownson RC, Fielding JE, Green LW. Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research. Annu Rev Public Health. 2018;39:27–53.

Brownson RC, Colditz GA, Proctor EK. Future issues in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. Second Edition ed. New York: Oxford University Press; 2018.

Thomson K, Hillier-Brown F, Todd A, McNamara C, Huijts T, Bambra C. The effects of public health policies on health inequalities in high-income countries: an umbrella review. BMC Public Health. 2018;18(1):869.

Download references

Acknowledgements

The authors are grateful for the policy expertise and guidance of Alexandra Morshed and the administrative support of Mary Adams, Linda Dix, and Cheryl Valko at the Prevention Research Center, Brown School, Washington University in St. Louis. We thank Lori Siegel, librarian, Brown School, Washington University in St. Louis, for assistance with search terms and procedures. We appreciate the D&I contributions of Enola Proctor and Byron Powell at the Brown School, Washington University in St. Louis, that informed this review. We thank Russell Glasgow, University of Colorado Denver, for guidance on the overall review and pragmatic measure criteria.

This project was funded March 2019 through February 2020 by the Foundation for Barnes-Jewish Hospital, with support from the Washington University in St. Louis Institute of Clinical and Translational Science Pilot Program, NIH/National Center for Advancing Translational Sciences (NCATS) grant UL1 TR002345. The project was also supported by the National Cancer Institute P50CA244431, Cooperative Agreement number U48DP006395-01-00 from the Centers for Disease Control and Prevention, R01MH106510 from the National Institute of Mental Health, and the National Institute of Diabetes and Digestive and Kidney Diseases award number P30DK020579. The findings and conclusions in this paper are those of the authors and do not necessarily represent the official positions of the Foundation for Barnes-Jewish Hospital, Washington University in St. Louis Institute of Clinical and Translational Science, National Institutes of Health, or the Centers for Disease Control and Prevention.

Author information

Authors and affiliations.

Prevention Research Center, Brown School, Washington University in St. Louis, One Brookings Drive, Campus Box 1196, St. Louis, MO, 63130, USA

Peg Allen, Meagan Pilar, Callie Walsh-Bailey, Stephanie Mazzucca, Maura M. Kepper & Ross C. Brownson

School of Social Work, Brigham Young University, 2190 FJSB, Provo, UT, 84602, USA

Cole Hooley

Kaiser Permanente Washington Health Research Institute, 1730 Minor Ave, Seattle, WA, 98101, USA

Cara C. Lewis, Kayne D. Mettert & Caitlin N. Dorsey

Department of Health Management & Policy, Drexel University Dornsife School of Public Health, Nesbitt Hall, 3215 Market St, Philadelphia, PA, 19104, USA

Jonathan Purtle

Brown School, Washington University in St. Louis, One Brookings Drive, Campus Box 1196, St. Louis, MO, 63130, USA

Ana A. Baumann

Department of Surgery (Division of Public Health Sciences) and Alvin J. Siteman Cancer Center, Washington University School of Medicine, 4921 Parkview Place, Saint Louis, MO, 63110, USA

Ross C. Brownson

You can also search for this author in PubMed   Google Scholar

Contributions

Review methodology and quality assessment scale: CCL, KDM, CND. Eligibility criteria: PA, RCB, CND, KDM, SM, MP, JP. Search strings and terms: CH, PA, MP with review by AB, RCB, CND, CCL, MMK, SM, KDM. Framework selection: PA, AB, CH, MP. Abstract screening: PA, CH, MMK, SM, MP. Full-text screening: PA, CH, MP. Pilot extraction: PA, DNC, CH, KDM, SM, MP. Data extraction: MP, CWB. Data aggregation: MP, CWB. Writing: PA, RCB, JP. Editing: RCB, JP, SM, AB, CD, CH, MMK, CCL, KM, MP, CWB. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Peg Allen .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

The authors declare they have no conflicting interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: table s1.

. PRISMA checklist. Table S2 . Electronic search terms for databases searched through EBSCO. Table S3 . Electronic search terms for searches conducted through PROQUEST. Table S4: PAPERS Pragmatic rating scales. Table S5 . PAPERS Psychometric rating scales.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Allen, P., Pilar, M., Walsh-Bailey, C. et al. Quantitative measures of health policy implementation determinants and outcomes: a systematic review. Implementation Sci 15 , 47 (2020). https://doi.org/10.1186/s13012-020-01007-w

Download citation

Received : 24 March 2020

Accepted : 05 June 2020

Published : 19 June 2020

DOI : https://doi.org/10.1186/s13012-020-01007-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation science
• Health policy
• Policy implementation
• Implementation
• Public policy
• Psychometric

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Program Finder
• Admissions Services
• Course Directory
• Academic Calendar
• Hybrid Campus
• Lecture Series
• Convocation
• Strategy and Development
• Implementation and Impact
• Integrity and Oversight
• In the School
• In the Field
• In Baltimore
• Resources for Practitioners
• Articles & News Releases
• In The News
• Statements & Announcements
• At a Glance
• Student Life
• Strategic Priorities
• Inclusion, Diversity, Anti-Racism, and Equity (IDARE)
• What is Public Health?
• Quantitative Methods in Public Health Certificate Program

OFFERED BY:  Office of Academic Affairs

Onsite | Part-Time | 1 - 3 years

• MAS Application Fee Waiver Requirements
• Master of Arts (MA) in Geography and Environmental Engineering
• Master of Arts and Master of Science in Public Health (MA/MSPH)
• Master of Arts in Public Health Biology (MAPHB)
• Master of Bioethics (MBE)
• Mission, Vision, and Values
• Student Experience
• Program Outcomes
• For Hopkins Undergraduate Students
• Master of Health Science (MHS) - Department of Biochemistry and Molecular Biology
• Master of Health Science (MHS) - Department of Epidemiology
• Alumni Update
• MHS Combined with a Certificate Program
• Master of Health Science (MHS) - Department of Molecular Microbiology and Immunology
• Alumni Highlights
• Post-Baccalaureate Program in Environmental Health for Pre-Medicine Students
• Bachelor's/MHS in Health Economics and Outcomes Research
• MHS HEOR Careers
• Frequently Asked Questions
• Master of Health Science (MHS)
• Concurrent School-Wide Master of Health Science Program in Biostatistics
• Master of Health Science - Department of Population, Family and Reproductive Health
• Master of Health Science Online (MHS) - Department of Population, Family and Reproductive Health
• Careers in Health Economics
• Core Competencies
• Meet the Director
• What is Health Economics
• MPH Capstone Schedule
• Concentrations
• Online/Part-Time Format
• Requirements
• Tuition and Funding
• Executive Board Faculty
• Master of Science (MS) in Geography and Environmental Engineering
• Independent Professional Project and Final Essay 
• Program Objectives and Outcomes
• Internships
• Master of Science (ScM) - Department of Biochemistry and Molecular Biology
• Master of Science (ScM) - Department of Biostatistics
• Master of Science (ScM) - Department of Epidemiology
• Master of Science (ScM) - Department of Molecular Microbiology and Immunology
• ScM Faculty Advisers
• Master of Science in Engineering (MSE) in Geography and Environmental Engineering
• Bachelor's/MSPH in Health Policy
• FAQ for MSPH in Health Policy
• Field Placement Experience
• MSPH Capstone
• MSPH Practicum
• Required and Elective Courses
• Student Timeline
• Career Opportunities
• 38-Week Dietetics Practicum
• Completion Requirements
• MSPH/RD Program FAQ
• Program Goals
• Master's Essay Titles
• Application Fee Waiver Requirements
• Doctor of Philosophy (PhD) - Department of Biostatistics
• Doctor of Philosophy (PhD) - Department of Epidemiology
• Program Goals and Expectations
• Doctor of Philosophy (PhD) - Department of Molecular Microbiology and Immunology
• Doctor of Philosophy (PhD) - Department of Population, Family and Reproductive Health
• Doctor of Philosophy (PhD) in Clinical Investigation
• Track in Environmental Sustainability, Resilience, and Health
• Track in Exposure Sciences and Environmental Epidemiology
• Track in Health Security
• Track in Toxicology, Physiology and Molecular Mechanisms
• PhD in Geography and Environmental Engineering Faculty Advisers
• Recent Graduates and Dissertation Titles
• PhD Funding
• PhD TA Requirement
• Recent Dissertation Titles
• JHU-Tsinghua Doctor of Public Health
• Core Course Requirements
• Concentration in Women’s and Reproductive Health
• Custom Track
• Concentration in Environmental Health
• Concentration in Global Health: Policy and Evaluation
• Concentration in Health Equity and Social Justice
• Concentration in Health Policy and Management
• Concentration in Implementation Science
• Meet Current Students
• Combined Bachelor's / Master's Programs
• Concurrent MHS Option for BSPH Doctoral Students
• Concurrent MSPH Option for JHSPH Doctoral students
• Doctor of Medicine and Doctor of Philosophy (MD/PhD)
• Adolescent Health Certificate Program
• Bioethics Certificate Program
• Climate and Health Certificate Program
• Clinical Trials Certificate Program
• Community- Based Public Health Certificate Program
• Demographic Methods Certificate Program
• Environmental and Occupational Health Certificate Program
• Epidemiology for Public Health Professionals Certificate Program
• Evaluation: International Health Programs Certificate Program
• Food Systems, the Environment and Public Health Certificate Program
• Frequently Asked Questions for Certificate Programs
• Gender and Health Certificate Program
• Gerontology Certificate Program
• Global Digital Health Certificate Program
• Global Health Certificate Program
• Global Health Practice Certificate Program
• Health Communication Certificate Program
• Health Disparities and Health Inequality Certificate Program
• Health Education Certificate Program
• Health Finance and Management Certificate Program
• Health and Human Rights Certificate Program
• Healthcare Epidemiology and Infection Prevention and Control Certificate Program
• Humane Sciences and Toxicology Policy Certificate Program
• Humanitarian Health Certificate Program
• Implementation Science and Research Practice Certificate Program
• Injury and Violence Prevention Certificate Program
• International Healthcare Management and Leadership Certificate Program
• Leadership for Public Health and Healthcare Certificate Program
• Lesbian, Gay, Bisexual, Transgender, and Queer (LGBTQ) Public Health Certificate Program
• Maternal and Child Health Certificate Program
• Mental Health Policy, Economics and Services Certificate Program
• Non-Degree Students General Admissions Info
• Pharmacoepidemiology and Drug Safety Certificate Program
• Population Health Management Certificate Program
• Population and Health Certificate Program
• Product Stewardship for Sustainability Certificate Program
• Public Health Advocacy Certificate Program
• Public Health Economics Certificate Program
• Public Health Informatics Certificate Program
• Public Health Practice Certificate Program
• Declaration of Intent - Public Health Preparedness
• Public Health Training Certificate for American Indian Health Professionals
• Public Mental Health Research Certificate Program
• Quality, Patient Safety and Outcomes Research Certificate Program
• Requirements for Successful Completion of a Certificate Program
• Rigor, Reproducibility, and Responsibility in Scientific Practice Certificate Program
• Risk Sciences and Public Policy Certificate Program
• Spatial Analysis for Public Health Certificate Program
• Training Certificate in Public Health
• Tropical Medicine Certificate Program
• Tuition for Certificate Programs
• Vaccine Science and Policy Certificate Program
• Online Student Experience
• Online Programs for Applied Learning
• Barcelona Information
• Fall Institute Housing Accommodations
• Participating Centers
• Registration, Tuition, and Fees
• Agency Scholarship Application
• General Scholarship Application
• UPF Scholarship Application
• Course Evaluations
• Online Courses
• Registration
• General Institute Tuition Information
• International Students
• Directions to the Bloomberg School
• All Courses
• Important Guidance for ONSITE Students
• D.C. Courses
• Registration and Fees
• Cancellation and Closure Policies
• Application Procedures
• Career Search
• Current Activities
• Current Trainees
• Related Links
• Process for Appointing Postdoctoral Fellows
• Message from the Director
• Program Details
• Admissions FAQ
• Current Residents
• Elective Opportunities for Visiting Trainees
• What is Occupational and Environmental Medicine?
• Admissions Info
• Graduates by Year
• Compensation and Benefits
• How to Apply
• Academic Committee
• Course Details and Registration
• Tuition and Fees
• ONLINE SOCI PROGRAM
• Principal Faculty
• Johns Hopkins RAPID Psychological First Aid
• General Application
• JHHS Application
• Areas of Study
• Important Dates
• Our Faculty
• Welcome Letter
• Descripción los Cursos
• Programa en Epidemiología para Gestores de Salud, Basado en Internet
• Consultants
• Britt Dahlberg, PhD
• Joke Bradt, PhD, MT-BC
• Mark R. Luborsky, PhD
• Marsha Wittink, PhD
• Rebekka Lee, ScD
• Su Yeon Lee-Tauler, PhD
• Theresa Hoeft, PhD
• Vicki L. Plano Clark, PhD
• Program Retreat
• Mixed Methods Applications: Illustrations
• Announcements
• 2023 Call for Applications
• Jennifer I Manuel, PhD, MSW
• Joke Bradt, PhD
• Josiemer Mattei, PhD, MPH
• Justin Sanders, MD, MSc
• Linda Charmaran, PhD
• Nao Hagiwara, PhD
• Nynikka R. A. Palmer, DrPH, MPH
• Olayinka O. Shiyanbola, BPharm, PhD
• Sarah Ronis, MD, MPH
• Susan D. Brown, PhD
• Tara Lagu, MD, MPH
• Theresa Hoft, PhD
• Wynne E. Norton, PhD
• Yvonne Mensa-Wilmot, PhD, MPH
• A. Susana Ramírez, PhD, MPH
• Animesh Sabnis, MD, MSHS
• Autumn Kieber-Emmons, MD, MPH
• Benjamin Han, MD, MPH
• Brooke A. Levandowski, PhD, MPA
• Camille R. Quinn, PhD, AM, LCSW
• Justine Wu, MD, MPH
• Kelly Aschbrenner, PhD
• Kim N. Danforth, ScD, MPH
• Loreto Leiva, PhD
• Marie Brault, PhD
• Mary E. Cooley, PhD, RN, FAAN
• Meganne K. Masko, PhD, MT-BC/L
• PhuongThao D. Le, PhD, MPH
• Rebecca Lobb, ScD, MPH
• Allegra R. Gordon, ScD MPH
• Anita Misra-Hebert, MD MPH FACP
• Arden M. Morris, MD, MPH
• Caroline Silva, PhD
• Danielle Davidov, PhD
• Hans Oh, PhD
• J. Nicholas Dionne-Odom, PhD RN ACHPN
• Jacqueline Mogle, PhD
• Jammie Hopkins, DrPH, MS
• Joe Glass, PhD MSW
• Karen Whiteman, PhD MSW
• Katie Schultz, PhD MSW
• Rose Molina, MD
• Uriyoán Colón-Ramos, ScD MPA
• Andrew Riley, PhD
• Byron J. Powell, PhD, LCSW
• Carrie Nieman MD, MPH
• Charles R. Rogers, PhD, MPH, MS, CHES®
• Emily E. Haroz, PhD
• Jennifer Tsui, Ph.D., M.P.H.
• Jessica Magidson, PhD
• Katherine Sanchez, PhD, LCSW
• Kelly Doran, MD, MHS
• Kiara Alvarez, PhD
• LaPrincess C. Brewer, MD, MPH
• Melissa Radey, PhD, MA, MSSW
• Sophia L. Johnson, PharmD, MPH, PhD
• Supriya Gupta Mohile, MD, MS
• Virginia McKay, PhD
• Andrew Cohen, MD, PhD
• Angela Chen, PhD, PMHNP-BC, RN
• Christopher Salas-Wright, PhD, MSW
• Eliza Park MD, MS
• Jaime M. Hughes, PhD, MPH, MSW
• Johanne Eliacin, PhD, HSPP
• Lingrui Liu ScD MS
• Meaghan Kennedy, MD
• Nicole Stadnick, PhD, MPH
• Paula Aristizabal, MD
• Radhika Sundararajan, MD
• Sara Mamo, AuD, PhD
• Tullika Garg, MD MPH FACS
• Allison Magnuson, DO
• Ariel Williamson PhD, DBSM
• Benita Bamgbade, PharmD, PhD
• Christopher Woodrell MD
• Hung-Jui (Ray) Tan, MD, MSHPM
• Jasmine Abrams, PhD
• Jose Alejandro Rauh-Hain, MD
• Karen Flórez, DrPH, MPH
• Lavanya Vasudevan, PhD, MPH, CPH
• Maria Garcia, MD, MPH
• Robert Brady, PhD
• Saria Hassan, MD
• Scherezade Mama, DrPH
• Yuan Lu, ScD
• 2021 Scholars
• Sign Up for Our Email List
• Workforce Training
• Cells-to-Society Courses
• Course/Section Numbers Explained
• Pathway Program with Goucher College
• The George G. Graham Lecture

About the Quantitative Methods in Public Health Certificate Program

The Quantitative Methods in Public Health certificate program offers participants an introduction to Epidemiology and Biostatistics. Upon completion of the program, participants will be able to evaluate the methods used to measure health effects in populations; interpret basic, quantitative public health measures; judge policy implications of public health data and research; be familiar with the difficulties of collecting, interpreting and analyzing data and their implications; and perform a critical review of public health/epidemiologic literature.

Curriculum for the Quantitative Methods in Public Health Certificate Program

Please visit our Academic Catalogue to see the full certificate curriculum requirements. Please also review the certificate program's completion requirements .

Admissions Requirements

Degree students.

Individuals who are currently matriculated in a degree program in the Johns Hopkins Bloomberg School of Public Health or other schools at the Johns Hopkins University are not eligible for this certificate program.

Non-Degree Students

Applicants are required to apply to certificate programs using SOPHAS Express . 

Eligible Start Terms :

Prerequisites or special requirements.

The certificate program is open to non-JHU students with at least a baccalaureate degree from an accredited college or university who have completed at least one college level Biology course and one college level Math or Statistics course with a grade of B or above, and a strong overall record of successful academic performance.

Application for Non-Degree Students

Information regarding the cost of tuition and fees can be found on the Bloomberg School's Certificate Programs Tuition page .

Financial Aid Eligibility: U.S. citizens and U.S. permanent residents enrolled in this certificate program may be eligible to apply for Title IV financial aid. Please contact the  JHU Office of Student Enrollment and Account Management (SEAM) for more information.

Questions about the program? We're happy to help.

Certificate Program Contact Rebecca Scott

Faculty Sponsor Sukon Kanchanaraksa, PhD Faculty Co-Sponsor Marie Diener-West, PhD

Language selection

• Français fr

Original quantitative research – Disparities in positive mental health of sexual and gender minority adults in Canada

HPCDP Journal Home

Submit a manuscript

• Information for authors

About HPCDP

• About the Journal
• Contact the Editors
• Past issues

Table of Contents | Next

Sonia Hajo, MSc Author reference footnote 1 Author reference footnote 2 ; Colin A. Capaldi, PhD Author reference footnote 1 ; Li Liu, PhD Author reference footnote 1

https://doi.org/10.24095/hpcdp.44.5.01

This article has been peer reviewed.

Recommended Attribution

Research article  by Hajo S et al. in the HPCDP Journal licensed under a Creative Commons Attribution 4.0 International License

Colin A. Capaldi, Centre for Surveillance and Applied Research, Public Health Agency of Canada, 785 Carling Ave, Ottawa, ON  K1A 0K9; Tel: 613-299-7714; Email: [email protected]

Hajo S, Capaldi CA, Liu L. Disparities in positive mental health of sexual and gender minority adults in Canada. Health Promot Chronic Dis Prev Can. 2024;44(5):197-207. https://doi.org/10.24095/hpcdp.44.5.01

Introduction: The goal of this study was to examine potential disparities in positive mental health ( PMH ) among adults in Canada by sexual orientation and gender modality.

Methods: Using 2019 Canadian Community Health Survey ( CCHS ) Annual Component data (N = 57 034), we compared mean life satisfaction and the prevalence of high self-rated mental health ( SRMH ), happiness and community belonging between heterosexual and sexual minority adults, and between cisgender and gender minority adults. We used 2019 CCHS Rapid Response on PMH data (N = 11 486) to compare the prevalence of high psychological well-being between heterosexual and sexual minority adults. Linear and logistic regression analyses examined the between-group differences in mean life satisfaction and the other PMH outcomes, respectively.

Results: Sexual minority (vs. heterosexual) adults reported lower mean life satisfaction ( B  = −0.7, 95% CI: −0.8, −0.5) and were less likely to report high SRMH ( OR = 0.4, 95% CI: 0.3, 0.5), happiness ( OR = 0.4, 95% CI: 0.3, 0.5), community belonging ( OR = 0.6, 95% CI: 0.5, 0.7) and psychological well-being ( OR = 0.4, 95% CI: 0.3, 0.6). Differences were not always significant for specific sexual minority groups in sex-stratified analyses. Gender minority adults reported lower mean life satisfaction and were less likely to report high SRMH and happiness than cisgender adults.

Conclusion: Future research could investigate how these PMH disparities arise, risk and protective factors in these populations, how other sociodemographic factors interact with sexual orientation and gender identity to influence PMH and changes in disparities over time.

Keywords : sexual orientation, gender identity, health inequalities, positive mental health, life satisfaction, happiness, psychological well-being, community belonging

• We investigated disparities in positive mental health ( PMH ) between sexual minority and heterosexual adults and between gender minority and cisgender adults in Canada in 2019.
• Mean life satisfaction was significantly lower among sexual minority adults than among heterosexual adults.
• Prevalence of high self-rated mental health, happiness, community belonging and psychological well-being were also significantly lower among sexual minority adults than among heterosexual adults.
• Mean life satisfaction and prevalence of high self-rated mental health and happiness were also significantly lower among gender minority adults than among cisgender adults.

Introduction

In 2015–2018, 3.2% of individuals in Canada aged 15 years and older identified as gay, lesbian or bisexual, Footnote 1 while in 2021, 0.3% identified as transgender or nonbinary. Footnote 2 Sexual orientation and gender modality (i.e. the congruence or incongruence between gender identity and sex assigned at birth Footnote 3 ) are sociodemographic characteristics that can have wide-ranging implications for health. Footnote 4 Footnote 5 Research shows that Two-Spirit, lesbian, gay, bisexual, transgender, queer, and additional people who identify as part of sexual and gender diverse communities ( 2SLGBTQ+ ) Footnote * individuals are at greater risk of negative psychological outcomes compared to their heterosexual and cisgender peers, Footnote 6 Footnote 7 including higher prevalence rates of depression and anxiety disorders, and of suicidal ideation and attempts among sexual minority Footnote 6 Footnote 8 Footnote 9 Footnote 10 and transgender Footnote 7 individuals. Non-suicidal self-injury is also more prevalent among sexual and gender minority ( SGM ) people. Footnote 11 Disparities in negative psychological outcomes between sexual minority and heterosexual people have also been observed in Canadian population health surveys. Footnote 12 Footnote 13 Footnote 14 Footnote 15 Footnote 16

These inequalities are often explained using minority stress theory—namely, that SGM people experience worse mental health on average due to the excess stress caused by the stigma, prejudice and discrimination they face and by the internalization of negative societal attitudes. Footnote 17 Footnote 18 Supporting this theory, stigmatizing events, internalized transphobia or homophobia, expectations of rejection and identity concealment have been associated with negative psychological outcomes among 2SLGBTQ+ individuals. Footnote 19 Footnote 20 Footnote 21 Footnote 22

While existing research may provide insights into the experience of mental illness and distress in 2SLGBTQ+ populations, these outcomes do not encompass all aspects of mental health. The World Health Organization defines health as “a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity.” Footnote 23 This definition implies that mental health spans beyond mental illness (i.e. ill -being) and emphasizes the importance of the positive aspects of mental health (i.e. well -being). Based on the dual-continuum model of mental health, positive mental health ( PMH ) and mental illness do not fall on opposing ends of a single continuum but are distinct (albeit related) constructs. Footnote 24 Footnote 25 Accordingly, an individual may live with a mental illness, but still have relatively high PMH . Footnote 26 Thus, to fully understand the mental health of SGM people, it is important to also examine their emotional, psychological and social well-being. Footnote 27 A focus on PMH can move us beyond traditional biomedical and deficit-based approaches to a more strengths-based understanding of the mental health of SGM individuals. Footnote 28

Some previous studies have investigated the overall PMH of sexual minority people in Canada using data from large population health surveys. Footnote 13 Footnote 29 Footnote 30 For instance, analyses of data from the 2012 Canadian Community Health Survey ( CCHS ) – Mental Health indicates that sexual minority individuals had lower PMH than heterosexual individuals, Footnote 29 Footnote 30 while analyses from the 2015 CCHS only showed significant disparities in PMH among bisexual individuals. Footnote 13 These studies examined PMH as a single broad construct in analyses; however, multiple PMH outcomes can be investigated to obtain a more fine-grained and nuanced understanding of different aspects of well-being in and between populations. Footnote 31 Footnote 32 Footnote 33 Indeed, the Public Health Agency of Canada ( PHAC ) monitors the PMH of adults in Canada using five outcomes in its Positive Mental Health Surveillance Indicator Framework ( PMH SIF): self-rated mental health ( SRMH ), happiness, life satisfaction, psychological well-being and community belonging. Footnote 31 Footnote 32

More generally, little is known about the PMH of gender minority individuals in Canada given that questions distinguishing between sex at birth and gender identity only began to be included in more recent population health surveys. Footnote 34

Finally, there are indications that some PMH outcomes like high SRMH have been decreasing in prevalence in Canada since 2015; Footnote 35 inequalities in PMH outcomes may have changed if temporal trends were not identical in both 2SLGBTQ+ and non- 2SLGBTQ+ populations. To address these gaps, we used more recent data from 2019 to comprehensively examine disparities in PMH by sexual orientation and by gender modality across PMH outcomes from the PMH SIF. Footnote 31 Footnote 32

Data and participants

Data for four out of five PMH outcomes came from the 2019 CCHS Annual Component, Footnote 34 which was collected from January to December 2019. The fifth PMH outcome, psychological well-being, was measured in the Rapid Response component of the 2019 CCHS , Footnote 36 administered to respondents who participated in January to March 2019. Statistics Canada excluded from the target population of the 2019 CCHS full-time members of the Canadian Armed Forces as well as individuals living on First Nations reserves and other Indigenous settlements in the provinces, in institutions, in foster care if aged 12 to 17 years, or in two specific health regions in Quebec; less than 3% of individuals in Canada aged 12 years and older are represented in these exclusions. Individuals living in the territories were excluded from the target population for the Rapid Response component; territorial data were collected but unavailable in the 2019 Annual Component as these data are only representative after 2 years of data collection. The sampling frame used for the Labour Force Survey was also used in the 2019 CCHS for adults living in the provinces.

Respondents completed the 2019 CCHS via computer-assisted telephone or in-person interviews. The 2019 CCHS collected data from individuals aged 12 years and older, although only those aged 15 years and older were asked about their sexual orientation. Nevertheless, we excluded youth aged 12 to 17 years from our study because different measures are used to monitor some indicators of PMH in the youth version of PMH SIF. Footnote 32 Moreover, we only had access to data from respondents who agreed to share their data with PHAC and Health Canada. As illustrated in Figure 1 , these restrictions led to sample sizes of 57 034 for the Annual Component and 11 486 for the Rapid Response.

This figure shows the screening process beginning with the number of scope units and ending with the number of individuals who are at least 18 years old.

Positive mental health outcomes

Of the five PMH outcomes included in the adult PMH SIF, Footnote 32 Footnote 33 SRMH , happiness, life satisfaction and community belonging were captured in the Annual Component. Psychological well-being was only measured in the Rapid Response. Our coding of high PMH was based on the cut-offs used in the PMH SIF. Footnote 32

SRMH was assessed with the question, “In general, would you say your mental health is…?” Response options included “excellent,” “very good,” “good,” “fair” and “poor.” We dichotomously coded individuals who responded “excellent” or “very good” as having high SRMH . This type of question has been recommended as a measure of general mental health status by the OECD. Footnote 25 Responses to this question have been associated with a wide range of physical and mental health outcomes. Footnote 37

Happiness was assessed with the question, “Would you describe yourself as being usually...?” Response options included “happy and interested in life,” “somewhat happy,” “somewhat unhappy,” “unhappy with little interest in life” and “so unhappy, that life is not worthwhile.” We dichotomously coded individuals who responded “happy and interested in life” as having high levels of happiness.

Life satisfaction was assessed with the question, “Using a scale of 0 to 10, where 0 means ‘very dissatisfied’ and 10 means ‘very satisfied,’ how do you feel about your life as a whole right now?” In the current research we treated this as a numerical variable and report on mean life satisfaction. Happiness and life satisfaction are core aspects of hedonic well-being or the positive feeling component of PMH . Footnote 27 Footnote 38

Community belonging was assessed with the question, “How would you describe your sense of belonging to your local community? Would you say it is…?” Response options included “very strong,” “somewhat strong,” “somewhat weak” and “very weak.” We dichotomously coded individuals who responded “very strong” or “somewhat strong” as having high community belonging. This question captures the social integration aspect of social well-being, which can be considered part of eudaimonic well-being or the positive functioning component of PMH (along with psychological well-being). Footnote 27 Footnote 38

Psychological well-being was measured using the six items from the psychological well-being subscale of the Mental Health Continuum—Short Form. Footnote 39 Respondents were asked how often in the past month they felt (1) that they liked most parts of their personality; (2) good at managing the responsibilities of their daily life; (3) that they had warm and trusting relationships with others; (4) that they had experiences that challenged them to grow and become a better person; (5) confident to think or express their own ideas and opinions; and (6) that their life had a sense of direction or meaning to it. These questions are designed to measure the six components of psychological well-being identified by Ryff Footnote 40 : self-acceptance, environmental mastery, positive relations with others, personal growth, autonomy and purpose in life. We recoded the following response options to represent the number of days in the past month: “every day“ (28 days = 7 days per week × 4 weeks); “almost every day” (20 days = 5 days per week × 4 weeks); “about 2 or 3 times a week” (10 days = 2.5 days per week × 4 weeks); “about once a week” (4 days = 1 day per week × 4 weeks); “once or twice” (1.5 days) and “never” (0 days). Footnote 41 We averaged the recoded responses and dichotomously coded respondents with a mean score of 20 or higher as having high psychological well-being.

Sexual orientation

Respondents were asked, “What is your sexual orientation?” Response options were “heterosexual,” “homosexual,” “bisexual” and “please specify.” Individuals who specified a sexual orientation that could be classified as one of the existing response options were recoded into that category by Statistics Canada. We coded individuals who identified as homosexual (gay/lesbian), bisexual/pansexual or another sexual orientation as a sexual minority.

Gender modality

Respondents were asked, “What was your sex at birth?” Response options included “male” and “female.” This was followed by the question, “What is your gender?” Response options included “male,” “female” and “please specify.” When responses for sex at birth and gender were the same, we coded the individual as cisgender; when responses differed, we coded the individual as a gender minority.

Using Annual Component data, we estimated mean life satisfaction and the percentage of high SRMH , happiness and community belonging by sexual orientation (heterosexual or sexual minority) and gender modality (cisgender or gender minority). We also obtained overall and sex-stratified estimates of these PMH outcomes for specific sexual minority groups (i.e. gay/lesbian and bisexual/pansexual); we did not separately report on PMH outcomes among those who identified as having another sexual orientation beyond heterosexual, gay/lesbian or bisexual/pansexual given the difficulty in interpreting findings for such a heterogeneous group.

We estimated the percentage of high psychological well-being using the Rapid Response data for individuals by sexual orientation (heterosexual or sexual minority). We also obtained overall and sex-stratified estimates of high psychological well-being for specific sexual minority groups (i.e. gay/lesbian and bisexual/pansexual). The estimate of psychological well-being for gender minority adults is not reported because it was not releasable (i.e. coefficient of variation [ CV ] > 35).

To determine whether the above estimates were significantly different, we conducted logistic regression analyses for the dichotomized PMH outcomes and linear regression analyses for life satisfaction. We used dummy coding for the linear regression analyses so that—similar to the logistic regression analyses—“heterosexual adults” was the reference group in the sexual orientation analyses and “cisgender adults” was the reference group in the gender identity analyses. We interpreted odds ratios ( OR ) with 95% confidence intervals ( CIs ) that did not include 1.0 as statistically significant in the logistic regression analyses. We interpreted coefficients with 95% CIs that did not include zero as statistically significant in the linear regression analyses.

For the overall comparisons of sexual minority adults to heterosexual adults, we also statistically controlled for a number of sociodemographic characteristics in follow-up logistic regression analyses for the dichotomized PMH outcomes and linear regression analyses for life satisfaction. Covariates included the individual’s sex at birth, age group, marital status (married/common law, single/never married, divorced/widowed/separated), highest educational attainment (high school or lower, postsecondary), racialized background (yes, no) and household income quintile.

In line with other analyses, Footnote 42 Footnote 43 Footnote 44 we coded the age of adults into four groups: young adults (18–34 years), younger middle-aged adults (35–49 years), older middle-aged adults (50–64 years) and older adults (65+ years). We grouped marital status and highest educational attainment into broad categories following previous analyses Footnote 45 and given the size of the sexual minority groups. There were minor discrepancies in how we coded racialized background due to different derived variables provided by Statistics Canada in each dataset at the time of analysis (see the Table 1 notes for more information).

Household income data were obtained by Statistics Canada from linked tax records, imputations or self-reports. Consistent with recommendations from Statistics Canada and given that income can have a nonlinear association with PMH outcomes, Footnote 46 we coded the household income values into quintiles. We also included place of residence (population centre, rural area) as a covariate in the analyses except for the one involving psychological well-being because it was not provided as a derived variable in the Rapid Response dataset by Statistics Canada. Population centres were defined by Statistics Canada as continuously built-up areas with populations of 1000+ and densities of 400+ per km Footnote 2 . Due to small sample sizes, we do not report follow-up logistic or linear regression analyses that control for covariates in comparisons involving specific sexual minority groups or gender minority adults.

All estimates were adjusted using sampling weights provided by Statistics Canada and variance was estimated using the bootstrap resampling method with 1000 replications. The sampling weights take into account non-response during the recruitment phase and non-sharing of data with PHAC and Health Canada by respondents. We dealt with missing data by using pairwise deletion to maximize the sample size for each analysis. Estimates with CVs between 15% and 35% (flagged with an “E” ) should be interpreted with caution due to high sampling variability; estimates with CVs above 35% (flagged with an “F” ) are suppressed. Analyses were conducted in SAS Enterprise Guide version 7.1 ( SAS Institute, Cary, NC , USA ).

Based on the Annual Component data, 0.2% E of adults in the Canadian provinces in 2019 were a gender minority and 3.9% were a sexual minority, with 1.7% identifying as gay/lesbian, 2.0% as bisexual/pansexual and 0.2% as another sexual orientation ( Table 1 ).

Sexual orientation and PMH

Sexual minority adults reported lower mean life satisfaction ( B = −0.7, 95% CI: −0.8, −0.5) and were less likely to report high SRMH ( OR = 0.4, 95% CI: 0.3, 0.5), high levels of happiness ( OR = 0.4, 95% CI: 0.3, 0.5), high community belonging ( OR = 0.6, 95% CI: 0.5, 0.7) and high psychological well-being ( OR = 0.4, 95% CI: 0.3, 0.6) than heterosexual adults ( Table 2 ). These differences were statistically significant even after controlling for covariates.

Overall, gay/lesbian and bisexual/pansexual adults reported significantly lower mean life satisfaction ( B = −0.4, 95% CI: −0.6, −0.2; B = −0.9, 95% CI: −1.1, −0.7, respectively) and were significantly less likely to report high SRMH ( OR = 0.7, 95% CI: 0.5, 0.9; OR = 0.3, 95% CI: 0.2, 0.4, respectively), high levels of happiness ( OR = 0.6, 95% CI: 0.4, 0.8; OR = 0.3, 95% CI: 0.2, 0.4, respectively), high community belonging ( OR = 0.6, 95% CI: 0.5, 0.8; OR = 0.6, 95% CI: 0.4, 0.7, respectively) and high psychological well-being ( OR = 0.4, 95% CI: 0.2, 0.8 E ; OR = 0.5, 95% CI: 0.3, 0.7, respectively) than heterosexual adults ( Table 3 ).

Significant differences across those five PMH outcomes were observed for both bisexual males and bisexual females in the sex-stratified analyses. Gay males were significantly less likely than heterosexual males to report high SRMH ( OR = 0.7, 95% CI: 0.5, 0.9) and community belonging ( OR = 0.5, 95% CI: 0.4, 0.7), but significant disparities were not observed for high levels of happiness, high psychological well-being or mean life satisfaction. In contrast, compared to heterosexual females, lesbian females reported significantly lower mean life satisfaction ( B = −0.6, 95% CI: −0.9, −0.3) and were significantly less likely to report high levels of happiness ( OR = 0.5, 95% CI: 0.3, 0.7); however, significant disparities were not found for high SRMH , community belonging or psychological well-being ( Table 3 ).

Gender modality and PMH

Gender minority adults reported significantly lower mean life satisfaction ( B = −1.7, 95% CI: −2.6, −0.9) and were significantly less likely to report high SRMH ( OR = 0.2, 95% CI: 0.1, 0.5) E and high levels of happiness ( OR = 0.2, 95% CI: 0.1, 0.4) E than cisgender adults, but a significant disparity in high community belonging was not observed ( Table 4 ).

This study documents the PMH of SGM adults across numerous outcomes in Canada in 2019, and investigates disparities in these PMH outcomes compared to heterosexual and cisgender adults. Overall, inequalities in PMH were common. Sexual minority adults reported lower mean life satisfaction and were less likely to report high SRMH , high levels of happiness, high community belonging and high psychological well-being compared to heterosexual adults. Similarly, gender minority adults had lower odds of reporting high SRMH and high levels of happiness, and tended to be less satisfied with life than cisgender adults.

These inequalities tended to be relatively large in magnitude when compared to disparities in PMH outcomes previously observed for other sociodemographic characteristics. Footnote 45 For instance, the percentage difference in high SRMH was 21.4 for sexual minority (vs. heterosexual) individuals and 36.0 E for gender minority (vs. cisgender) individuals in the current study, while the percentage difference in high SRMH did not exceed 14.1 in 2019 for comparisons by age group, racialized group membership, immigrant status, household income, place of residence, educational attainment, parental status, living alone, marital status, official language minority or Indigenous identity in previous analyses. Footnote 45 Moreover, the difference in mean life satisfaction was 0.7 for sexual minority (vs. heterosexual) individuals and 1.7 for gender minority (vs. cisgender) individuals in the current study, while the highest mean difference in all the sociodemographic comparisons listed above was 0.6 in 2019. Footnote 45 The especially sizable inequalities in PMH in SGM populations identify a high priority for mental health promotion activities, as well as other interventions aimed at addressing potential determinants of PMH . Footnote 47 Footnote 48

Beyond these overall inequalities, it is also important to acknowledge the heterogeneity that exists within SGM groups. Although PMH tended to be less prevalent among SGM individuals compared to heterosexual and cisgender individuals, there were still large portions of SGM individuals who reported high levels of PMH . For example, high community belonging was reported by the majority of gay, lesbian, bisexual/pansexual and gender minority individuals. Investigations into risk and protective factors that distinguish SGM individuals who report high PMH from those who do not could be important for understanding and promoting individual and community resilience in these populations. Footnote 49 For instance, greater self-compassion appears to be a protective factor as it has been linked to lower minority stress and better well-being among SGM populations. Footnote 50 In contrast, SGM people in Canada are more likely to report experiencing violent victimization, Footnote 51 which is a risk factor of lower PMH ; Footnote 31 Footnote 32 Footnote 33 safer and more 2SLGBTQ+ friendly communities are likely an important social determinant for these populations and a potential target for more systemic-level interventions. Footnote 47

There were differences in the consistency by which inequalities in PMH outcomes were observed in this study, with disparities between bisexual/pansexual versus heterosexual adults being the most robust. This is in line with previous research findings that the risk of negative psychological outcomes is often highest for bisexual individuals compared to heterosexual or gay/lesbian individuals. Footnote 6 Footnote 8 Footnote 10 Footnote 11 Footnote 52 The distinctive prejudice and discrimination that can be experienced by bisexual people has been offered as an explanation for their heightened risk, including the negative societal attitudes about bisexuality, the invisibility and erasure of bisexual people in wider society, and the lack of affirmative support for bisexual individuals. Footnote 52 Indeed, a recent environmental scan only found one program in Canada that was exclusively dedicated to addressing the social determinants of health among bisexual persons. Footnote 47

Disparities in PMH also tended to be prominent for gender minority adults. Beyond community belonging, only around one-third of gender minority adults reported high SRMH and high levels of happiness, and they rated their life satisfaction 1.7 points lower, on average, than did cisgender individuals. These findings expand previous research on the prevalence of negative psychological outcomes in the transgender population. Footnote 7 Footnote 11 Reducing distal stressors (i.e. being the target of transphobic behaviours) and proximal stressors (i.e. expectations of rejection or discrimination, transgender identity concealment and internalized transphobia) could be important for mental health promotion, as these experiences have been associated with depression and suicidal ideation among gender minority individuals. Footnote 19 Future research could explore risk and protective factors of PMH for gender minority people.

Strengths and limitations

By using data from large population health surveys, we were able to investigate numerous PMH outcomes in the overall SGM populations as well as in specific sexual minority groups. The examination of PMH among gender minority individuals is an especially important contribution as the inclusion of questions asking about both sex and gender is a recent development in Statistics Canada surveys. In addition, our strengths-based focus on PMH allowed us to document that—despite population disparities—many SGM individuals report experiencing well-being in their lives.

Nevertheless, there are limitations that warrant mention. First, we identified many disparities in PMH and offered potential explanations for the results based on minority stress theory and previous research, but we did not directly examine why the disparities exist. The inequalities across the PMH outcomes persisted for sexual minority adults compared to heterosexual adults when we statistically controlled for various sociodemographic characteristics; however, distinct groups were broadly coded into one category for some of the covariates and only unadjusted regression analyses for PMH outcomes were reported for comparisons involving specific sexual minority groups and gender minority adults due to small sample sizes. The small sample sizes also resulted in some relatively wide CIs and likely affected the statistical power to detect significant differences. In addition, the small number of gender minority adults in the dataset restricted our ability to examine specific gender identities (e.g. transgender men, transgender women, nonbinary individuals). The oversampling of SGM individuals in future population health surveys could allow for more comprehensive examinations of specific SGM identities, as well as the disaggregation of results by other potentially important sociodemographic factors. Footnote 53 For instance, age breakdowns could be informative; experiences of discrimination and disparities in mental health have been found to vary across the life course among sexual minority individuals in other countries. Footnote 54 Footnote 55

Self-reported responses to questionnaires may be subject to recall bias and social desirability bias. Footnote 56 Further, the unwillingness of some respondents to disclose their sexual orientation or gender modality could have resulted in some misclassification. Footnote 57 While respondents were asked to report on their sexual identity, there are other important dimensions of sexual orientation that could have been assessed (i.e. sexual attraction and sexual behaviour). Footnote 53 Footnote 58 Moreover, the survey question on gender included “male” and “female” as response options instead of the more relevant “man” and “woman.” Footnote 59 Finally, we may be missing data from the most at-risk SGM individuals (e.g. those who are experiencing homelessness). Footnote 60

We found that PMH tended to be less common among SGM adults than among heterosexual and cisgender adults in 2019. Future research could explore the mechanisms by which SGM people experience lower PMH , risk and protective factors of PMH in SGM populations, how PMH might depend on the interaction between sexual orientation and gender modality with other sociodemographic characteristics, and how the observed disparities in PMH may have changed over time.

Acknowledgements

We would like to thank Raelyne Dopko and Elia Palladino for their feedback in the early stages of planning this project, and Mélanie Varin for her feedback at multiple stages of the project and for re-running analyses to double-check results.

Conflicts of interest

The authors have no conflicts of interest.

Authors’ contributions and statement

• SH : Writing – original draft, writing – review & editing.
• CAC : Conceptualization, methodology, writing – original draft, writing – review & editing.
• LL : Methodology, formal analysis, writing – review & editing.

All authors approved the manuscript for publication.

The content and views expressed in this article are those of the authors and do not necessarily reflect those of the Government of Canada.

Page details

A .gov website belongs to an official government organization in the United States.

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• 10 Essential Public Health Services
• Cooperative Agreements, Grants & Partnerships
• Public Health Professional: Programs
• Health Assessment: Index
• Research Summary
• COVID-19 Health Disparities Grant Success Stories Resources
• Communication Resources

Performance Management and Quality Improvement: Definitions and Concepts

At a glance.

In the public health field, many initiatives and organizations focus on improving public health practice, using different terms. This page provides common definitions for public health performance management.

Definitions and concepts

There has been a rapidly growing interest in performance and quality improvement within the public health community, and different names and labels are often used to describe similar concepts or activities. Other sectors, such as industry and hospitals, have embraced a diverse and evolving set of terms but which generally have the same principles at heart (i.e., continuous quality improvement, quality improvement, performance improvement, six sigma, and total quality management).

In the public health field, an array of initiatives has set the stage for attention to improving public health practice, using assorted terms. The Turning Point Collaborative focused on performance management, the National Public Health Performance Standards Program created a framework to assess and improve public health systems, while the US Department of Health and Human Services has provided recommendations on how to achieve quality in healthcare . In 2011, the Public Health Accreditation Board launched a national voluntary accreditation program that catalyzes quality improvement but also acknowledges the importance of performance management within public health agencies. Regardless of the terminology, a common thread has emerged—one that focuses on continuous improvement and operational excellence within public health programs, agencies, and the public health system.

To anchor common thinking, below are links to some of the definitions that are frequently used throughout these pages.

Key definitions

• Riley et al, "Defining Quality Improvement in Public Health", JPHMP, 2010, 16(10), 5-7.
• Public Health Accreditation Board Acronyms and Glossary of Terms, Version 2022 [PDF]

Public Health Gateway

CDC's National Center for State, Tribal, Local, and Territorial Public Health Infrastructure and Workforce helps drive public health forward and helps HDs deliver services to communities.

ORIGINAL RESEARCH article

Quantitative evaluation of the medicine innovation policy in china: based on the pmc-index model.

• Shanghai International College of Intellectual Property, Tongji University, Shanghai, China

Introduction: Medicine innovation is crucial in promoting the sustainable development of medicine undertakings, which has significant economic and social benefits. China is the main force in global medicine consumption, with a huge demand for innovative medicines. Thus, the Chinese government releases a series of policies aimed at providing scientific and reasonable guidance for medicine innovation. However, there is inadequate quantitative evaluation and comparison of various medicine innovation policies in the existing studies.

Methods: This paper adopts the approach of text mining and the Policy Modeling Consistency Index (PMC-Index) model to construct an evaluation system and then quantitatively evaluates and compares the traditional Chinese medicine innovation policies (TCMIPs), the biological medicine innovation policies (BMIPs), and the multiple medicine innovation policies (MMIPs) in China.

Results: The results indicate that: (1) The three types of drug innovation policies have similarities in content and goal through comparative analysis of high-frequency words, while they also have their own characteristics. (2) The average PMC-Index of 29 TCMIPs is 5.77, which has the highest policy bad rate (21%); the average PMC-Index of 12 BMIPs is 6.21, which has the highest policy good rate (92%); moreover, the average PMC-Index of 35 MMIPs is 6.06, which has the highest policy excellence rate (26%). (3) The BMIPs, MMIPs, and TCMIPs have similar scores on policy object, policy orientation, policy timeliness, policy evaluation, and policy accessibility, while they differ significantly mainly on policy nature, incentive method, policy function, policy issuing agency, and policy instrument.

Discussion: This study contributes to a comprehensive understanding of medicine innovation policies in China, in order to provide theoretical support for future policy formulation and optimization in the medicine industry. Moreover, we expand the application scenarios of policy diffusion theory.

1 Introduction

Medicine plays an important role in safeguarding human health and promoting medical progress ( 1 ). Innovative medicine is the frontier force of the pharmaceutical industry, which has huge social and economic benefits ( 2 ). Cancer and autoimmune diseases have become major health challenges globally, while innovative medicines can improve the cure rate of these difficult diseases ( 3 ). In 2022, the global market size of innovative medicines reached 1,027 billion dollars, accounting for approximately 69.5% of the global pharmaceutical market. However, the scale of the innovative medicine market in China is only 142 billion dollars in 2022. Moreover, due to a lack of competition among similar high-quality medicines, the prices of imported innovative medicines have remained high for a long time in China. Before the domestic programmed death-1 was available, patients spent about 70,000 to 80,000 dollars annually on imported innovative medicines such as Keytruda and Opdivo ( 4 ). However, the treatment cost for patients decreases to less than 7,000 dollars when domestic innovative medicines launch. Domestic innovative medicines are gradually replacing imported medicines, which contributes to controlling health insurance expenditures and reducing the burden on patients. Furthermore, China is aging much faster than the global average. The proportion of older adult people aged 65 years and older in China doubled to 14.2% from 2000 to 2021 ( 5 ). More and more older adult people mean a growing market space for the innovative medicine industry, and the demand for innovative medicines will continue to increase in the future. How to stimulate medicine innovation in China is significant and urgent, so the Chinese government has formulated a large number of medicine innovation policies.

These medicine innovation policies include development plans, guidelines, and implementation opinions to support and encourage the development of the medicine industry. This indicates that the government is highly concerned about public health ( 6 , 7 ). Reviewing the medicine innovation policies in China, what are the similarities and differences among these policy texts? What are the overall quality and individual characteristics of the medicine innovation policy in China? How can we identify the strengths and weaknesses of medicine innovation policy design and provide targeted improvement strategies? Academics have yet to answer these questions. Thus, the motivation of this study is to assess medicine innovation policy in China, answer the above research questions, and help policymakers improve the medicine innovation system to promote medicine innovation development. However, the medicine innovation policy in China lacks a comprehensive and scientific method for evaluating the advantages and disadvantages of policies. There are various methods for policy evaluation ( 8 – 10 ), but the more cutting-edge at present is the PMC-Index model ( 11 ). The PMC-Index is a policy evaluation methodology proposed by Estrada in 2011 that assesses the internal consistency of policies in several dimensions and identifies the advantages and disadvantages of each policy ( 12 ). This paper attempts to use the PMC-Index method to quantitatively assess the consistency level of medicine innovation policies in China.

This study attempts to fill the gaps in the existing literature, and our major contributions are as follows: (1) There is insufficient comparative analysis of medicine policies in the existing literature, with some studies focusing on a single industry ( 13 ). However, this study conducts comparative analysis and selects comprehensive samples, including the traditional Chinese medicine innovation policies (TCMIPs), the biological medicine innovation policies (BMIPs), and the multiple medicine innovation policies (MMIPs) issued by the Chinese government; MMIPs cover the innovation of traditional Chinese medicine, biological medicine, and chemical medicine at the same time. Since the retrieved policies relate to the innovation of chemical medicines are technical guidelines without specific planning content, our study excludes policies that relate to chemical medicine innovation in the analysis. (2) Most of the existing literature is about the macro-evaluation of the implementation effect of medicine innovation policies, as it is the endpoint of policy evaluation ( 14 – 16 ), but neglects the analysis of policy content ( 17 – 19 ). However, text mining technology is adopted in this study to dig deeply into the policy texts of TCMIPs, BMIPs, and MMIPs so as to identify the basic elements and the internal logic of various medicine innovation policies. Moreover, this paper constructs the PMC-Index model to quantitatively evaluate and compare the TCMIPs, BMIPs, and MMIPs, respectively, which provides theoretical support for future policy formulation in the medicine industry. The research framework is illustrated in Figure 1 .

Figure 1 . Research framework.

The rest of this study is organized as follows: Section 2 reviews the literature on medicine innovation and policy evaluation, and identifies the shortcomings in existing research. Section 3 describes the research design, including sample selection and methodology. Section 4 reports the quantitative analysis results of TCMIPs, BMIPs, and MMIPs and proceeds with a comparative analysis and discussion of various policies. Section 5 summarizes this study and elaborates on the limitations.

2 Literature review

To promote better practice of medicine innovation policies, scholars have studied these policies from both quantitative and qualitative perspectives. The quantitative research regarding medicine innovation policies mainly focuses on their implementation effects evaluation, while the qualitative research regarding these policies chiefly concentrates on their policy content evaluation. Compared with the qualitative research, the quantitative research on medicine innovation policies is more abundant. That is, most existing literature focuses on the implementation effectiveness of medicine innovation policies. For example, Bouet ( 20 ) utilized probit and logit techniques to evaluate the effect of TRIPs on Indian pharmaceutical industry innovation. Gamba ( 21 ) employed the zero-inflated negative binomial model to analyze the medicine intellectual property protection reform policies in developed and developing countries and believed that medicine innovation is highly sensitive to intellectual property protection. Aghmiuni et al. ( 22 ) discussed the supportive innovation policies in biological medicine and found that these policies have a significant impact on the development of biological medicine innovation. Moreover, existing studies also indicate that some medicine innovation policies in China contribute to innovation quality. For example, Liu et al. ( 23 ) applied the difference-in-differences model to find that the generic consistency evaluation policy has a positive impact on the innovation quality of the Chinese pharmaceutical industry. Gu et al. ( 24 ) adopted listed medicine companies in the Shanghai and Shenzhen A-share markets as samples and empirically found that centralized drug procurement policy has a significant improvement on the quality of medicine innovation. In addition, the qualitative evaluation of medicine innovation policies in existing research mainly concentrates on the specific interpretation of policy content. Doran et al. ( 25 ) reviewed the reforms of medicine innovation policies in Australia and elaborated on the impact of price control on innovation. Karampli et al. ( 26 ) outlined the study findings on the impact of medicine innovation on medicine expenditure growth and described the challenges faced by Greek drug innovation policies. Liu et al. ( 27 ) described the medicine innovation policies to accelerate medicine review and approval in China, stating that the development of innovative medicine benefits from these accelerated policies. Overall, the above literature helps researchers understand medicine innovation policies from different perspectives and provides insights for policy optimization. However, these studies have mainly examined medicine innovation policies at the macro level, lacking a systematic evaluation of medicine innovation policies in China. Thus, research on medicine innovation policies in China has yet to be expanded.

Policy evaluation is a complex and systematic program that plays a crucial role in guiding policy formulation and optimization ( 28 – 30 ). Choosing appropriate and scientific evaluation methods is the foundation of policy evaluation. The existing studies have proposed a variety of methods to evaluate policies, such as the five kinds of evaluation tools ( 31 ), the “3e “assessment framework ( 32 ), the index of legal changes ( 33 ), hierarchical analysis ( 34 ), Delphi method ( 35 ), content analysis method ( 36 ), and difference in difference analysis ( 37 – 39 ). As previously stated, the five kinds of evaluation tools, the “3e “assessment framework, and the index of legal changes are relatively outdated and one-sided in their assessment of policy. Moreover, hierarchical analysis, the Delphi method, and the content analysis method have more subjective evaluation processes ( 40 ). Furthermore, the difference in difference analysis focuses on evaluating the implementation effect of a certain policy ( 41 – 43 ), lacking systematic evaluation of a series of policies ( 44 ). These above policy evaluation methods are widely adopted, but they fall short in terms of objectivity and accuracy. In addition, these policy evaluation methods pay less attention to individual differences and the texts of policies. However, the PMC-Index model combines qualitative and quantitative approaches in a more comprehensive and objective way than the above methods, which are widely used to evaluate policies. It can provide an overall evaluation of policy consistency as well as systematically analyze individual policy differences from various dimensions. In previous literature, research on the PMC-Index model has been on the rise, and many satisfactory results have been achieved. For example, Liu et al. ( 45 ) utilized the PMC-Index model to discuss the power battery recycling policies of the central and local governments and found that the policymaking ability of the central government is stronger. Fan et al. ( 46 ) employed the PMC-Index model to investigate China’s municipal solid waste policies and identified that these policies are generally reasonable. Zhao et al. ( 47 ) utilized the PMC-Index model to explore energy security in China and believed that the administrative level of the issuing agency positively affects the PMC-Index. In addition, many studies have used this model for policy evaluation, including fire safety education policy ( 48 ), traditional Chinese medicine development policy ( 13 ), new energy vehicle policy ( 49 , 50 ), and internet healthcare policy ( 51 ). These studies reflect that the PMC-Index model has good applicability for opening up the black box of policy formulation and promoting policy evaluation.

Overall, there is extensive literature on medicine innovation and policy evaluation, while the existing research still has some shortcomings. First, there is much literature on the implementation performance of medicine innovation policies, while little literature evaluates these policies from a policy formulation perspective. Second, the existing literature has not yet applied the PMC-Index model to evaluate medicine innovation policies, lacking the inclusion and comparative study of TCMIPs, BMIPs, and MMIPs simultaneously. Hence, this paper aims to narrow these gaps by investigating the TCMIPs, BMIPs, and MMIPs, constructing an evaluation indicator system, and utilizing the PMC-Index model to analyze these policies. The goal of this study is to gain insights into the current status of various medicine innovation policies and provide references for the formulation and improvement of these policies in the future.

3 Research design

3.1 data sources and samples selection.

The medicine innovation policies issued by the Chinese government are taken as the research object in this study. To obtain the policy texts on the medicine innovation policies systematically, we adopt three search paths. Firstly, relevant policy documents are retrieved on the portals of the State Council (SC), the National Health Commission (NHC), the National Medical Products Administration (NMPA), the National Administration of Traditional Chinese Medicine (NATCM), and other related government departments. Secondly, we search for relevant policy documents on the Peking University Law Website. 1 Finally, search platforms such as Baidu and Google are used as supplements for policy document collection. We set search terms such as “medicine innovation,” “TCM innovation,” “biological medicine innovation,” and “chemical medicine innovation” in these databases. Considering the evolution characteristics of China’s medicine innovation policies, the retrieval period is from 2000 to 2023. Due to some repeated and invalid collection, the policy documents are screened according to the following principles: (1) only the national-level policy documents are selected in this study; (2) we eliminate some documents that have been revised or repealed; (3) policy documents such as working arrangements, letters, technical guidelines, and approvals are excluded; and (4) we focus on policies with specific plans. After eliminating the irrelevant policies, 76 policy documents are obtained, including 29 TCMIPs, 12 BMIPs, and 35 MMIPs (some policies are shown in Table 1 ). These policy documents mainly cover laws, regulations, plans, outlines, notices, and other relevant rules on medicine innovation in China.

Table 1 . The 29 TCMIPs, 12 BMIPs, and 35 MMIPs (partial).

3.2 Identification of the policy text features

Before the construction of the PMC-Index model, ROSTCM 6 software is adopted for text mining of the above policies ( 45 , 48 ). We process the policy documents using the ROSTCM6 software, including policy integration, word segmentation, and high-frequency word statistics. Words that appear more frequently but are meaningless, such as “construct,” “increase,” and “development,” are deleted. Finally, the most relevant and frequent words are extracted for further analysis. High-frequency words can reflect the topic of general interest in policy documents ( 52 ). Besides, the top 30 high-frequency words are selected from TCMIPs, BMIPs, and MMIPs, and the Gephi software is adopted to establish a co-occurrence network to clearly show the difference and relevance of various types of medicine innovation policies.

3.3 Construction of the PMC-Index model

The PMC-Index model is a scientific and quantitative measurement method for policy evaluation. This model is proposed by Estrada ( 12 ), which originates from the Omnia Mobilis hypothesis. The hypothesis believes that everything is in motion and interconnected, so any seemingly irrelevant variable should not be ignored, and the quantity and weights of variables are not restricted. The PMC-Index model analyzes the advantages and disadvantages of each policy and the consistency level of a policy in multiple dimensions by selecting variables comprehensively ( 47 , 53 ). The PMC-Index model is composed of four main steps ( 52 , 54 ) (see Figure 2 ).

Figure 2 . Steps to construct the PMC-Index model.

3.3.1 Variable classification and parameter identification

The classification of variables and identification of parameters are essential bases for comprehensive policy evaluation. According to the existing studies ( 45 – 47 ) and the specific characteristics of medicine innovation policy, we establish 10 primary variables, namely policy nature (X 1 ), policy issuing agency (X 2 ), policy object (X 3 ), policy timeliness (X 4 ), policy instrument (X 5 ), policy orientation (X 6 ), incentive method (X 7 ), policy function (X 8 ), policy evaluation (X 9 ), and policy accessibility (X 10 ). The sub-variables are set for the primary variable by the relevant literature and policy text mining, as shown in Table 2 . After classifying, it is essential to identify the parameters of the variables. The binary method is adopted to assign equal weight to all sub-variables ( 48 , 55 ). If the policy content conforms to the sub-variable, the parameter is set to 1; otherwise, the parameter is set to 0 ( 46 ).

Table 2 . Evaluation index system of the medicine innovation policy.

3.3.2 Building a multi-input-output table

The multi-input-output table is an analysis framework capable of data storage that evaluates an individual variable in multiple dimensions ( 60 , 61 ). Building a multi-input-output table is a precondition for the PMC-Index calculation of the medicine innovation policy. In this study, the multi-input-output tables consist of 10 primary variables and 40 sub-variables. The primary variables are not specially ordered and are mutually independent of each other. The sub-variables refine the primary variables in different aspects. In this paper, all researchers analyze and determine whether the policy content involves the sub-variables, respectively. The evaluation results of all researchers are almost the same, apart from a few variables. The controversial variables are further analyzed and discussed collectively on the basis of the policy content and evaluation criteria. After parameter identification, the multi-input-output tables for 29 TCMIPs, 12 BMIPs, and 35 MMIPs are established, as shown in Table 3 .

Table 3 . The multi-input-output table for quantitative evaluation of XX.

3.3.3 Measurement of the PMC-Index

The PMC-Index calculation includes the specific four steps ( 46 , 55 ). Firstly, the primary variables and sub-variables are integrated into the multi-input-output tables of TCMIPs, BMIPs, and MMIPs, respectively. Secondly, the binary method is adopted to assign the value of each sub-variable according to text analysis and Eqs 1 , 2 . Thirdly, the values of 10 primary variables are calculated individually based on Eq. 3 . Fourthly, we sum up all primary variables to obtain the PMC-Index of policies according to Eq. 4 .

Where X i refers to the ith primary variable, i = 1, 2, 3, …, 10. X ij refers to the ijth sub-variable, j = 1, 2, 3, …, n . T ( Xij ) refers to the number of sub-variables of the ith primary variable.

The PMC indexes of 29 TCMIPs, 12 BMIPs, and 35 MMIPs are calculated based on the above steps. The PMC-Index can evaluate the comprehensiveness and degree of policy consistency. Due to the 10 primary variables selected in our evaluation system, the value of the PMC-Index should be [0, 10]. According to existing studies ( 12 , 52 , 55 ), we classify the values of PMC-Index into 4 evaluation levels (see Table 4 ).

Table 4 . Evaluation criteria for policy based on the PMC-Index.

3.3.4 Construction of the PMC-Surface

The PMC-Surface is constructed to visualize the strengths and weaknesses of policies in multiple dimensions. We plot the PMC-Surface by calculating the PMC matrix. To meet the balance and symmetry of the matrix, X 10 is left out of this research ( 11 , 46 , 47 ). After removing X 10 , a 3 × 3 matrix is generated by the remaining 9 primary variables, as shown in Eq. 5 . Then we utilize the above matrix to draw the PMC-Surface. The concave-convex degree and color depth of the PMC-Surface reflect the strengths and weaknesses of each policy visually. MATLAB software is applied to draw the PMC-Surface diagrams.

4 Results and analysis

4.1 analysis of high-frequency words.

The central zone surrounded by the high-frequency words reflects the common concern of the TCMIPs, BMIPs, and MMIPs. As shown in Figure 3 , it can be seen that “innovation” is located in the center area and has a high frequency, for “innovation” is the core theme of TCMIPs, BMIPs, and MMIPs. Table 5 illustrates that 30% of high-frequency words extracted from the TCMIPs, BMIPs, and MMIPs are the same, with shared highly-frequency words such as “science and technology,” “innovation,” “system,” “mechanism,” and “resource,” suggesting that the common focus of policy is on promoting science and technology development, allocating relevant resources, and improving the medicine innovation system.

Figure 3 . High-frequency words network of TCMIPs, BMIPs and MMIPs.

Table 5 . Statistics of the top 30 high frequency words from the TCMIPs, BMIPs and MMIPs.

The TCMIPs and BMIPs all involve high-frequency words such as “talent” and “capability,” indicating that talent cultivation is an important guarantee for the innovation of traditional Chinese medicine and biological medicine. The TCMIPs and MMIPs all involve high-frequency words such as “health,” “clinical,” and “institution,” suggesting that these policies emphasize the clinical efficacy of innovative medicines. The BMIPs and MMIPs all involve high-frequency words such as “enterprise,” “platform,” “treatment,” “disease,” and “safety,” which indicates that pharmaceutical enterprises are encouraged to build innovation platforms to develop efficient and safe innovative medicine. In addition, there is also important information at the edge region in Figure 3 , which should not be ignored and reflects the different concerns of TCMIPs, BMIPs, and MMIPs.

The dedicated high-frequency words in TCMIPs involve “nation,” “culture,” “civilization,” “protection,” “criteria,” “evaluation,” and “cooperation,” which indicates that the government emphasizes protection and inheritance for TCMI culture, construction evaluation criteria for TCMI standardized management, and greater international cooperation on TCMI. The dedicated high-frequency words in BMIPs include “economics,” “industrialization,” “strategy,” etc. Due to the small scale of most biopharmaceutical enterprises, they are encouraged to carry out an industrialization strategy to boost BMI development. The dedicated high-frequency words in MMIPs involve “approval,” “regulation,” “achievement,” and “market.” These high-frequency words illustrate that the MMIPs prefer to create a favorable external environment to promote medicine innovation, for example, by speeding up innovative medicine approval, strengthening medicine regulation, encouraging achievement transformation, and cultivating the market environment.

4.2 Index analysis and comparison of medicine innovation policy

4.2.1 index analysis of tcmips.

Based on the above evaluation system and criteria, we calculate the PMC-Index and determine the level of TCMIPs, as shown in Table 6 . The average PMC-Index of 29 TCMIPs is 5.77, which indicates good overall consistency in TCMIPs. Specifically, there are 23 TCMIPs with good consistency and 6 TCMIPs with bad consistency, while there are no excellent and no perfect among the 29 TCMIPs. In addition, these policies are mainly released by the NHC, the NMPA, and the NATCM, suggesting that the Chinese government attaches great importance to TCMI.

Table 6 . The PMC-Index and level of the medicine innovation policy.

With the development of the TCM industry, the focus of TCMIPs has shifted from a general outline (P2, P5) to a specific implementation plan (P20, P21, and P27). For instance, P20 proposes detailed promotion measures for TCM innovation, P21 focuses on medical insurance to support TCM innovation, and P27 emphasizes scientific supervision to stimulate TCM innovation. This reflects the tendency of TCMIP formulations to shift from “rough” to “refined,” which is more conducive to policy implementation ( 62 , 63 ). It has become one of the strategies emphasized for how to encourage the TCMI in the coming decades.

In this study, we select P2 (Good level) and P9 (Bad level) to display the differences between TCMIPs visually. The PMC-Surfaces of these selected TCMIPs are drawn according to the PMC matrix (see Figures 4A , B ). The convex surface means a higher score on the corresponding primary variable, whereas the concave surface indicates a lower score. The PMC-Index of P2 is 6.90, ranking first among the 29 TCMIPs. P2 is jointly issued by seven government departments, making relatively comprehensive arrangements to promote TCMI. As shown in Figure 4A , the surface shape of P2 is relatively smooth except for the X 6 (Policy orientation). This is because the policy orientation of P2 only involves encouragement and support, which neglects normative guidance and compulsory requirements. Due to the lower score of P2 on policy orientation, the improvement for P2 should take normative guidance and compulsory requirement on policy orientation into account.

Figure 4 . The PMC-Surface of (A) P2 and (B) P9 of TCMIPs.

The PMC-Index of P9 is 4.25, ranking 29th among the 29 TCMIPs. As shown in Figure 4B , there is obviously convexity in X8 of P9, while the overall surface shape of P9 is concave compared to P2. This indicates that P9 performs poorly on most of the primary variables. This is due to P9, which measures the registration of scientific and technological achievements in TCM and focuses mainly on the X8 (policy function) but ignores the other primary variables. The improvement path for policy is determined according to the difference between the primary variables and the average ( 47 – 49 ). Thus, the improvement path of P9 is X 9 -X 3 -X 1 -X 5 -X 7 -X 2 -X 6 .

4.2.2 Index analysis of BMIPs

As shown in Table 6 , we calculate the PMC-Index and determine the level of BMMIPs based on the above evaluation system and criteria. The average PMC-Index of 12 BMIPs is 6.21, which indicates good overall consistency in BMIPs. Moreover, these policies are mainly released by the SC, the National Development and Reform Commission (NDRC), and the Ministry of Science and Technology (MST), which reflects that the Chinese government attaches much significance to BMI. There are 1 BMIP with excellent consistency and 11 BMIPs with good consistency, while there are no bad or perfect BMIPs among the 12 BMIPs.

Q3 has excellent consistency due to five powerful primary variables: policy object (X 3 ), policy instrument (X 5 ), incentive method (X 7 ), policy function (X 8 ), and policy evaluation (X 9 ). Q3 is a notice on the biological “11th Five-Year Plan,” which is issued by the SC. This document covers a comprehensive range of policy objects, including government departments, medical institutions, enterprises, and scientific research institutions. Previous studies have divided policy instruments into supply side instruments (SSI), demand side instruments (DSI), and environmental side instruments (ESI) ( 45 , 59 ). The above three types of policy instruments are used in Q3, including SSI that directly supports BMI development, DSI that directly pulls BMI development, and ESI that creates a favorable external environment for BMI development. Q3 adopts tax benefits, investment subsidies, and intellectual property protection to inspire BMI. The function of Q3 involves five specific aspects of X 8 , which aim to systematically promote innovation in biological medicine. Thus, Q3 is well designed according to five strong primary variables.

Q3 (Excellent level) and Q2 (Good level) are selected to display the differences between BMIPs visually in this study. The PMC-Index of Q3 is 7.02 ranking first among the 12 BMIPs, while the PMC-Index of Q2 is 5.38 ranking 12th among the 12 BMIPs. The PMC-Surfaces of these selected BMIPs are drawn according to the PMC matrix, as shown in Figures 5A , B . It can be seen that the PMC-Surface of Q3 lies at a higher location than that of Q2, which indicates that Q3 has better consistency. However, Q2 is a special announcement and aims to provide funding for BMI, which is relatively single on the policy scope and has bad consistency. Except for policy timeliness (X4) and policy orientation (X 6 ), other primary variables in Q2 are lower than the average in different degrees; thereby, the improvement path of Q2 is X 8 -X 7 -X 3 -X 9 -X 5 -X 1 -X 2 .

Figure 5 . The PMC-Surface of (A) Q3 and (B) Q2 of BMIPs.

4.2.3 Index analysis of MMIPs

We calculate the PMC-Index and determine the level of MMIPs based on the above evaluation system and criteria, as shown in Table 6 . The average PMC-Index of 35 MMIPs is 6.06, which indicates good overall consistency in MMIPs. Specifically, there are 9 MMIPs with excellent consistency, 20 MMIPs with good consistency, 6 MMIPs with bad consistency, and no perfect MMIP among the 35 MMIPs. In addition, these policies are mainly released by the SC, the MST, the NHC, and the NMPA, suggesting that there is a higher authority in MMIP policy issuance.

As the medical industry develops, the MMIPs emphasis has shifted from a broad plan (R1, R4) to a detailed incentive scheme (R25, R32, and R34). For instance, R25 focuses on technology transfer to drive medicine innovation; R32 proposes detailed evaluation measures to accelerate innovative medicine to the market; and R34 emphasizes regulatory capacity construction to supervise innovative medicine development. Hence, the current MMIPs focus is to formulate practical and concrete implementation plans to promote medicine innovation.

The R33 (Excellent level) and R28 (Bad level) are selected to display the differences between MMIPs visually in this study. The PMC-Surfaces of these selected MMIPs are drawn according to the PMC matrix (see Figures 6A , B ). The PMC-Index of R33 is 7.88, ranking first among the 35 MMIPs. It can be seen that the PMC -Surface of R33 is overall convex, for there are four coordinate points at scale “1” in Figure 6A . This policy is jointly issued by nine government departments, manifesting sufficient coordination and cooperation among departments. R33 aims to clarify the primary goals of medical industry development and accelerate the improvement of the medicine innovation system during the 14th Five-Year Plan period. Overall, R33 has comprehensive content, complete support, and abundant policy instruments, which is a scientifically reasonable policy.

Figure 6 . The PMC-Surface of (A) R33 and (B) R28 of MMIPs.

The PMC-Index of R28 is 4.48, ranking 35th among the 35 MMIPs. As shown in Figure 6B , the PMC-Surface of R28 displays fewer convex points, while most surfaces are dented, which suggests that P2 scores lower on various variables. This is because R28 only adopts the means of legal constraints to guide medicine innovation without making specific arrangements in other aspects. The policy issuing agency is single, the policy instrument is deficient, and the policy function is narrow, leading to the dented surfaces of R28. Except for policy object (X 3 ) and policy timeliness (X 4 ), other primary variables in R28 are lower than the average in different degrees; thereby, the improvement path of R28 is X 8 -X 9 -X 5 -X 7 -X 2 -X 1 -X 6 .

4.2.4 Comparative analysis among TCMIPs, BMIPs, and MMIPs

To clarify the characteristics and differences of various medicine innovation policies in China, we compare the TCMIPs, BMIPs, and MMIPs for further analysis. Upon comparing the average PMC-Index of three types of medicine innovation policies, the order from high to low is BMIPs (6.21) > MMIPs (6.06) > TCMIPs (5.77), which suggests that BMIPs and MMIPs are superior to TCMIPs. In addition, the PMC-Index of TCMIPs range from 4.25 to 6.90, the PMC-Index of BMIPs range from 5.38 to 7.02, and the PMC-Index of MMIPs range from 4.48 to 7.88. In order to clearly display the distribution of the three types of policies across each evaluation level, we create the Table 7 . As shown in Table 7 , the MMIPs have the highest policy excellence rate (26%), the BMIPs have the highest policy good rate (92%), and the TCMIPs have the highest policy bad rate (21%). Next, we will explore the reasons for the differences through a comparative analysis of primary variables.

Table 7 . The proportion of TCMIPs, BMIPs, and MMIPs in various levels.

To show the differences of the various medicine innovation policies more intuitively, we construct a comparative radar chart of the average scores of the primary variables for BMIPs, MMIPs, and TCMIPs. As shown in Figure 7 , the BMIPs, MMIPs, and TCMIPs have similar scores on X 3 , X 4 , X 6 , X 9 , and X 10 , while they differ significantly mainly on X 1 , X 2 , X 5 , X 7 , and X 8 . The specific analysis is as follows:

(1) X 1 (Policy nature). Generally, policy nature is closely related to the purpose of policy formulation; in this paper, policy nature includes prediction, guidance, description, supervision, planning, and support ( 64 , 65 ). The policy nature of TCMIPs is relatively homogeneous, most only involving guidance and description. Compared with TCMIPs and MMIPs, most BMIPs cover multiple policy natures (i.e., prediction, guidance, description, plan, and support) simultaneously. There are 49% of MMIPs that contain four types of policy natures (i.e., prediction, guidance, description, and plan) at the same time, ranging between TCMIPs and BMIPs. Overall, the three types of medicine innovation policies have less involvement in supervision, which should be incorporated into future policy formulation.

(2) X 2 (policy issuing agency). The existing studies find that cooperation among policy issuing agencies affects the coordination ability of policy implementation ( 47 , 66 , 67 ). TCMIPs are mainly released independently or jointly by the NHC, the NMPA, and the NATCM, wherein 41% of TCMIPs are released independently by the NATCM and 24% of TCMIPs are released jointly by multiple departments. Moreover, BMIPs are mainly released by the SC, the NDRC, and the MST, whereas only 8% of BMIPs are released jointly by multiple departments. Further, MMIPs are mainly released by the SC, the MST, the NHC, and the NMPA, wherein 54% of MMIPs are released jointly by multiple departments. This suggests that, compared with the BMIPs and the TCMIPs, the MMIPs concentrate more on the coordination and cooperation among departments when formulating policies, which is more conducive to policy implementation.

(3) X 5 (policy instrument). As mentioned earlier, policy instruments are generally divided into SSI, ESI, and DSI ( 45 , 59 ). SSI mainly involves the support of financial, infrastructure construction, and technical information. ESI mainly involves regulation, supervision, and public opinion publicity. DSI mainly involves government procurement and service outsourcing. The policy instruments adopted by TCMIPs are SSI and ESI, while BMIPs and MMIPs cover three types of policy instruments. In addition, the usage frequency of DSI in BMIPs is slightly higher than that in MMIPs.

(4) X 7 (incentive method). Incentive methods refer to measures that promote policy implementation, such as tax benefits, investment subsidies, intellectual property protection, regulatory and evaluation, and administrative approval incentives. From the content of the policy text, some BMIPs and MMIPs adopt multiple incentive methods, while most TCMIPs only involve one incentive method. Thus, in terms of incentive methods, BMIPs and MMIPs are superior to TCMIPs. However, the average score of the incentive methods for BMIPs and MMIPs is 0.42, indicating that they are generally unsatisfactory and that there is much room for improvement.

(5) X 8 (policy function). The policy function means the social effect that can be achieved after policy implementation ( 68 , 69 ). In this paper, policy functions mainly include talent cultivation, encouraging innovations, industry-academia-research collaboration, achievement transformation, and international exchange. There are most BMIPs covering the above six functions, followed by TMIPs and MMIPs. It is worth noting that the average policy functions (X 7 ) score of MMIPs is 0.71, which indicates that the functions of medicine policies are generally good.

Figure 7 . Comparison of primary variables among TCMIPs, BMIPs, and MMIPs.

5 Discussion

Based on the above analysis results, we will further discuss the following aspects. First, we find that the nature of medicine innovation policy mainly focuses on guidance, description, and plan, which is similar to the findings of existing studies. For example, Zhang et al. ( 52 ) researched coal power policy and concluded that these policies are mainly manifested as guidance and description. Yang et al. ( 13 ) found that more than 90% of TCM development policies involved guidance and planning. This study, using a larger policy sample, responds to their results. Moreover, we have new findings compared with their research. We find that the nature of supervision is rarely incorporated in medicine innovation policies, and there is a lack of supervision over implementation effectiveness.

Second, there is an overall lack of cooperation among departments in this study from the perspective of the policy issuing agency. This is similar to some previous research findings, which also believe that there is weak collaboration among policy issuing agencies ( 11 , 46 , 62 ). This may be due to differences in specific responsibilities and potential competition among departments, which result in a reluctance to cooperate and coordinate with each other. However, existing research suggests that cross-departmental cooperation is more likely to enhance administrative efficiency ( 70 – 72 ). Moreover, many TCMIPs are independently issued by the NATCM, which is a vice-ministerial department with a relatively low administrative level. These findings are similar to those of Yang et al. ( 49 ). In general, the administrative level of the policy issuing agency has a positive impact on the efficiency of policy implementation ( 66 ). Furthermore, the two departments that are most relevant to the issuance of medicine innovation policies in China are the NMPA and the NATCM, which are under the State Administration for Market Regulation and the NHC, respectively. Hence, it is difficult to coordinate the management of these two departments and jointly formulate medicine innovation policies in practice.

Third, in terms of policy instruments, BMIPs and MMIPs perform well overall, while TCMIPs need to improve. Specifically, we find that the TCMIPs lack the application of DSI. This paper complements the study of Yang et al. ( 13 ), in which policy instruments are not included in the evaluation system. Moreover, Xiong et al. ( 48 ) analyzed the application of policy instruments in the fire safety education policy, involving voluntary policy tools, mandatory tools, and mixed policy tools. The classification of policy instrument types differs between Xiong and us, which belong to different schools. Overall, we have further enriched the research on policy instruments.

Finally, the incentive method for the medicine innovation policy is overall insufficient. Specifically, medicine innovation policy in China takes financial subsidies and intellectual property protection as the main incentive methods, while other incentive methods are seldom involved. Similar findings have been reported in previous studies. For example, Yang et al. ( 49 ) found that the new energy vehicle policy has inadequate incentives for infrastructure construction. Yang et al. ( 11 ) conducted a study on 37 health promotion policies that seldom involve incentives. These findings indicate that insufficient incentives are a common problem with policies. However, a lack of appropriate incentive methods will hinder the rapid growth of the industry.

6 Conclusions and implications

6.1 conclusion.

This study quantitatively evaluates and analyzes the medicine innovation policies in China since 2000 through mining the text and the PMC-Index model. Specifically, we summarize the respective characteristics of the TCMIPs, BMIPs, and MMIPs and further compare the similarities and differences of the three types of policies from horizontal perspectives. Then, further discussion and analysis are conducted, eliciting suggestions for medicine innovation policy improvement. As far as we know, this study is the first to quantitatively explore the consistency of medicine innovation policies across different types in China and fills the gap in existing literature.

Based on the analysis of the high-frequency words of medicine innovation policy, the results show that the common focus of policy is to promote the development of science and technology, allocate related resources, and improve the pharmaceutical innovation system. In addition, various medicine innovation policies also have their own unique focus. Specifically, the TCMIPs emphasize the inheritance and innovation of TCM; the BMIPs focus on encouraging biopharmaceutical companies to implement industrialization strategies; and the MMIPs tend to create a favorable external environment to promote medicine innovation.

This study utilizes the PMC-Index model to systematically evaluate the TCMIPs, BMIPs, and MMIPs, respectively. The results showed that the average PMC-Index of 29 TCMIPs is 5.77, wherein the policy at the good and bad levels accounts for 79 and 21%, respectively. Compared to BMIPs and MMIPs, the average primary variable values of policy nature (X 1 ), policy instrument (X 5 ), and incentive method (X 7 ) are relatively lower in TCMIPs, so priority could be given to improving TCMIPs in these areas in the future. The average PMC-Index of 12 BMIPs is 6.21, wherein the policy at excellent level accounts for 8% and the policy at good level accounts for 92%, respectively. Moreover, the average PMC-Index of 35 MMIPs is 6.06, wherein the policy at excellent level accounts for 26%, the policy at good level accounts for 57%, and the policy at bad level accounts for 17%, respectively. These results indicate that none of the medicine innovation policies in China reach a perfect consistency level, so there is still room for improvement. In addition, the BMIPs, MMIPs, and TCMIPs have similar scores on X 3 , X 4 , X 6 , X 9 , and X 10 , while they differ significantly mainly on X 1 , X 2 , X 5 , X 7 , and X 8 .

6.2 Implications

The theoretical significance of this study is reflected in several aspects. First, this study provides a theoretical reference for the medicine innovation policy formulation in China. In other words, policymakers can design more effective policies by drawing on our study when formulating medicine innovation policies in the future. Second, the PMC-Index model has not previously been used to evaluate medicine innovation policies. Thus, the PMC-Index model is utilized to evaluate the medicine innovation policy in this study, which enriches this stream of literature. Finally, we extend the application context of policy diffusion theory to evaluate medicine innovation policy. Specifically, this paper verifies that the formulation of medicine innovation policy in China conforms to the connotation of policy diffusion theory.

Based on the above analysis, the policy implications proposed in this study are as follows. (1) According to the different policy timeliness, corresponding assessment times should be set when formulating policies, such as long-term monitoring of more than 5 years, medium-term monitoring of 3–5 years, short-term monitoring of 1–3 years, and monitoring within 1 year. Then, establish monitoring standards based on different innovation policy objectives and monitor when the assessment time is reached. Through the establishment of a cross-departmental monitoring information sharing platform, all departments can share their supervision results of medicine innovation policies. Based on the results of supervisory feedback, problems existing in medicine innovation policies should be optimized when formulating policies in the future. (2) The joint superior department of the NMPA and the NATCM can be set up when institutional reform is carried out in the future. This joint superior department plays a connecting role among the SC, other ministerial-level departments, the NMPA, and the State Administration of TCM. The joint superior department can coordinate multiple departments to release medicine innovation policies based on the goals and contents of the policies, improve the administrative level of policy release, and thereby promote medicine innovation. (3) In the formulation of BMIPs and MMIPs, the three policy instruments should be further integrated to bring into play the complementarity among them. Government departments should refer to the application of DSI in BMIPs and MMIPs and incorporate DSI (i.e., government procurement and government purchasing) into the formulation of future TCMIPs, taking into account the characteristics of TCMIPs. DSI can directly stimulate policy receptors and produce significant effects. (4) Incentives should be appropriately enriched and diversified to provide sufficient incentives for medicine innovation in policy formulation. Provide the implementation details of the corresponding incentives, such as the application conditions of the incentive method, to prevent the abuse of the incentives.

6.3 Limitations and further research

This study investigates the strengths and weaknesses of medicine innovation policies from the perspective of policy formulation, which provides new insights and a theoretical basis for the evaluation of medicine innovation policies in the future. However, some limitations in our research should be realized. First, there may be some subjectivity in the variable identification. To obtain a more objective and scientific policy evaluation, we can optimize the shortcomings in future research through various methods such as grounded theory, crawler mining, and big data methods. Second, this study mainly selects medicine innovation policies from the national level while not including the local level. In future studies, medicine innovation policies issued by local governments can be included for comparative analysis because they have local characteristics and provide richer references for policy formulation. Finally, this study only evaluates the text of medicine innovation policies in China without analyzing the performance of policy implementation. The difference in difference econometric analysis model can be used to further analyze the implementation effect of medicine innovation policies in the future.

Data availability statement

The original contributions presented in the study are included in the article/ Supplementary materials , further inquiries can be directed to the corresponding author.

Author contributions

DG: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Writing – original draft. LQ: Data curation, Formal analysis, Methodology, Supervision, Writing – review & editing. XS: Conceptualization, Data curation, Formal analysis, Funding acquisition, Resources, Supervision, Writing – review & editing.

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This research was funded by the National Social Science Foundation of China (16ZDA236).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2024.1403320/full#supplementary-material

Abbreviations

PMC-Index, Policy Modeling Consistency Index; TCM, Traditional Chinese medicine; TCMIPs, Traditional Chinese medicine innovation policies; BMIPs, Biological medicine innovation policies; MMIPs, Multiple medicine innovation policies; SSI, Supply side instruments; ESI, Environmental side instruments; DSI, Demand side instruments; SC, State Council; NHC, National Health Commission; NMPA, National Medical Products Administration; NATCM, National Administration of traditional Chinese medicine; NDRC, National Development and Reform Commission; MST, Ministry of Science and Technology.

1. ^ http://www.pkulaw.cn/

1. Debaveye, S, De Soete, W, De Meester, S, Vandijck, D, Heirman, B, Kavanagh, S, et al. Human health benefits and burdens of a pharmaceutical treatment: discussion of a conceptual integrated approach. Environ Res . (2016) 144:19–31. doi: 10.1016/j.envres.2015.10.027

Crossref Full Text | Google Scholar

2. Kanavos, P, Visintin, E, and Gentilini, A. Algorithms and heuristics of health technology assessments: a retrospective analysis of factors associated with HTA outcomes for new drugs across seven OECD countries. Soc Sci Med . (2023) 331:116045. doi: 10.1016/j.socscimed.2023.116045

3. Pe, M, Alanya, A, Falk, RS, Amdal, CD, Bjordal, K, Chang, J, et al. Setting international standards in analyzing patient- reported outcomes and quality of life endpoints in cancer clinical trials-innovative medicines initiative (SISAQOL-IMI) stakeholder views, objectives, and procedures. Lancet Oncol . (2023) 24:E270–83. doi: 10.1016/s1470-2045(23)00157-2

PubMed Abstract | Crossref Full Text | Google Scholar

4. Chen, Y. How to boost long-term confidence in China’s innovative drugs. China Reform . (2023) 1:59–65.

Google Scholar

5. Chen, XX, Giles, J, Yao, Y, Yip, WN, Meng, QQ, Berkman, L, et al. The path to healthy ageing in China: a Peking University lancet commission. Lancet . (2022) 400:1967–2006. doi: 10.1016/s0140-6736(22)01546-x

6. Li, Y, Gao, Y, Meng, X, Liu, X, and Feng, Y. Assessing the air pollution abatement effect of prefabricated buildings in China. Environ Res . (2023) 239:117290. doi: 10.1016/j.envres.2023.117290

7. Shen, Q, Pan, YX, and Feng, YC. Identifying impacts of industrial co-agglomeration on carbon emissions: evidence from China. Front Public Health . (2023) 11:1154729. doi: 10.3389/fpubh.2023.1154729

8. Gao, X, Zhang, G, Zhang, Z, Wei, Y, Liu, D, and Chen, Y. How does new energy demonstration city pilot policy affect carbon dioxide emissions? Evidence from a quasi-natural experiment in China. Environ Res . (2024) 244:117912. doi: 10.1016/j.envres.2023.117912

9. Shen, Q, Pan, Y, Meng, X, Ling, X, Hu, S, and Feng, Y. How does the transition policy of mineral resource-exhausted cities affect the process of industrial upgrading? New empirical evidence from China. Resour Policy . (2023) 86:104226. doi: 10.1016/j.resourpol.2023.104226

10. Sun, CW, Xu, SH, Yang, M, and Gong, X. Urban traffic regulation and air pollution: a case study of urban motor vehicle restriction policy. Energ Policy . (2022) 163:112819. doi: 10.1016/j.enpol.2022.112819

11. Yang, YL, Tang, J, Li, ZX, and Wen, J. How effective is the health promotion policy in Sichuan, China: based on the PMC-index model and field evaluation. BMC Public Health . (2022) 22:2391. doi: 10.1186/s12889-022-14860-9

12. Estrada, MAR. Policy modeling: definition, classification and evaluation. J Policy Model . (2011) 33:523–36. doi: 10.1016/j.jpolmod.2011.02.003

13. Yang, CR, Yin, SC, Cui, D, Mao, ZF, Sun, Y, Jia, CL, et al. Quantitative evaluation of traditional Chinese medicine development policy: a PMC index model approach. Front Public Health . (2022) 10:1041528. doi: 10.3389/fpubh.2022.1041528

14. Jiang, YT, Xiao, YN, Zhang, ZH, and Zhao, S. How does central-local interaction affect local environmental governance? Insights from the transformation of central environmental protection inspection in China. Environ Res . (2024) 243:117668. doi: 10.1016/j.envres.2023.117668

15. Niu, SH, Chen, YD, Zhang, RW, Luo, RF, and Feng, YC. Identifying and assessing the global causality among energy poverty, educational development, and public health from a novel perspective of natural resource policy optimization. Resour Policy . (2023) 83:103770. doi: 10.1016/j.resourpol.2023.103770

16. Zhang, ZH, Shang, YZ, Zhang, GX, Shao, S, Fang, JY, Li, PX, et al. The pollution control effect of the atmospheric environmental policy in autumn and winter: evidence from the daily data of Chinese cities. J Environ Manag . (2023) 343:118164. doi: 10.1016/j.jenvman.2023.118164

17. Chun, N, and Park, M. The impact of health insurance mandates on drug innovation: evidence from the United States. Eur J Health Econ . (2013) 14:323–44. doi: 10.1007/s10198-012-0379-6

18. Um, SI, Sohn, UD, Jung, SY, You, SH, Kim, C, Lee, S, et al. Longitudinal study of the impact of three major regulations on the Korean pharmaceutical industry in the last 30 years. Health Res Policy Syst . (2022) 20:4. doi: 10.1186/s12961-021-00797-9

19. Wu, GY, Sun, MM, and Feng, YC. How does the new environmental protection law affect the environmental social responsibility of enterprises in Chinese heavily polluting industries? Hum Soc Sci Commun . (2024) 11:1–14. doi: 10.1057/s41599-024-02674-6

20. Bouet, D. A study of intellectual property protection policies and innovation in the Indian pharmaceutical industry and beyond. Technovation . (2015) 38:31–41. doi: 10.1016/j.technovation.2014.10.007

21. Gamba, S. The effect of intellectual property rights on domestic innovation in the pharmaceutical sector. World Dev . (2017) 99:15–27. doi: 10.1016/j.worlddev.2017.06.003

22. Aghmiuni, SK, Siyal, S, Wang, Q, and Duan, YZ. Assessment of factors affecting innovation policy in biotechnology. J Innov Knowl . (2020) 5:179–89. doi: 10.1016/j.jik.2019.10.002

23. Liu, Q, Huang, Z, and Mao, ZB. Has the consistency evaluation policy of generic drugs promoted the innovation quality of Chinese pharmaceutical manufacturing industry? An empirical study based on the difference-in-differences model. Front Public Health . (2023) 11:1265756. doi: 10.3389/fpubh.2023.1265756

24. Gu, Y, and Zhuang, Q. Does China’s centralized volume-based drug procurement policy facilitate the transition from imitation to innovation for listed pharmaceutical companies? Empirical tests based on double difference model. Front Pharmacol . (2023) 14:1192423. doi: 10.3389/fphar.2023.1192423

25. Doran, E, and Henry, DA. Australian pharmaceutical policy: Price control, equity, and drug innovation in Australia. J Public Health Policy . (2008) 29:106–20. doi: 10.1057/palgrave.jphp.3200170

26. Karampli, E, Souliotis, K, Polyzos, N, Kyriopoulos, J, and Chatzaki, E. Pharmaceutical innovation: impact on expenditure and outcomes and subsequent challenges for pharmaceutical policy, with a special reference to Greece. Hippokratia . (2014) 18:100–6.

27. Liu, Y, Zhang, N, Xie, CC, Jiang, YL, Qin, YH, Zhou, LY, et al. Evolution of drug regulations and regulatory innovation for anticancer drugs in China. Acta Pharm Sin B . (2022) 12:4365–77. doi: 10.1016/j.apsb.2022.08.004

28. Dommarco, JAR, de Cosío, TG, García-Chávez, CG, and Colchero, MA. The role of public nutrition research organizations in the construction, implementation and evaluation of evidence-based nutrition policy: two National Experiences in Mexico. Nutrients . (2019) 11:594. doi: 10.3390/nu11030594

29. Führ, M, and Bizer, K. REACh as a paradigm shift in chemical policy: responsive regulation and behavioural models. J Clean Prod . (2007) 15:327–34. doi: 10.1016/j.jclepro.2005.10.003

30. Zhang, JX, Cheng, C, and Feng, YC. The heterogeneous drivers of CO2 emissions in China’s two major economic belts: new evidence from spatio-temporal analysis. Environ Dev Sustain . (2023) 26:10653–10679. doi: 10.1007/s10668-023-03169-1

31. Bigman, SK. Evaluative research: principles and practice in public service and social action programs. By Edward A Suchman. J Leis Res . (1969) 1:209–11. doi: 10.1080/00222216.1969.11969732

32. OF Poland. Program evaluation and administrative theory. Public Adm Rev . (1974) 34:333–8. doi: 10.2307/975243

33. Libecap, GD. Economic variables and the development of the law: the case of Western mineral rights. J Econ Hist . (1978) 38:338–62. doi: 10.1017/S0022050700105121

34. Saaty, TL. A new macroeconomic forecasting and policy evaluation method using the analytic hierarchy process. Math Model . (1987) 9:219–31. doi: 10.1016/0270-0255(87)90479-9

35. Deloe, RC. Exploring complex policy questions using the policy delphi- a multi round, interactive survey method. Appl Geogr . (1995) 15:53–68. doi: 10.1016/0143-6228(95)91062-3

36. Foote, SB, Halpern, R, and Wholey, DR. Variation in Medicare’s local coverage policies: content analysis of local medical review policies. Am J Manag Care . (2005) 11:181–7.

37. Zhang, SJ, Dong, R, Jiang, JX, Yang, SY, Cifuentes-Faura, J, Peng, SH, et al. Whether the green credit policy effectively promote green transition of enterprises in China? Empirical analysis and mechanism verification. Environ Res . (2024) 244:117910. doi: 10.1016/j.envres.2023.117910

38. Yang, M, Liu, X, and Li, Z. Can the construction of digital infrastructure promote labor force employment? — evidence from the pilot policy of “broadband China”. Sys Eng Theory Pract . (2024) 44:190–206. doi: 10.12011/SETP2023-1579

39. Wu, G, Gao, Y, and Feng, Y. Assessing the environmental effects of the supporting policies for mineral resource-exhausted cities in China. Resour Policy . (2023) 85:103939. doi: 10.1016/j.resourpol.2023.103939

40. Cahuc, P, and Le Barbanchon, T. Labor market policy evaluation in equilibrium: some lessons of the job search and matching model. Labour Econ . (2010) 17:196–205. doi: 10.1016/j.labeco.2009.08.012

41. Ling, X, Luo, ZW, Feng, YC, Liu, X, and Gao, Y. How does digital transformation relieve the employment pressure in China? Empirical evidence from the national smart city pilot policy. Hum Soc Sci Commun . (2023) 10:1–17. doi: 10.1057/s41599-023-02131-w

42. Hua, C, Zhang, Z, Miao, J, Sun, H, and Jia, F. Do urban agglomeration planning policies promote the discharge reduction of industrial wastewater: evidence from the Yellow River Basin in China. Environ Res . (2023) 239:117445. doi: 10.1016/j.envres.2023.117445

43. Niu, SH, Chen, YD, Zhang, RW, and Feng, YC. How does the air pollution prevention and control action plan affect sulfur dioxide intensity in China? Front Public Health . (2023) 11:1119710. doi: 10.3389/fpubh.2023.1119710

44. Vogel, M, Kakani, P, Chandra, A, and Conti, RM. Medicare price negotiation and pharmaceutical innovation following the inflation reduction act. Nat Biotechnol . (2024) 42:406–12. doi: 10.1038/s41587-023-02096-w

45. Liu, F, Tang, J, Rustam, A, and Liu, Z. Evaluation of the central and local power batteries recycling policies in China: a PMC-index model approach. J Clean Prod . (2023) 427:139073. doi: 10.1016/j.jclepro.2023.139073

46. Fan, XH, Chu, ZJ, Chu, X, Wang, S, Huang, WC, and Chen, JC. Quantitative evaluation of the consistency level of municipal solid waste policies in China. Environ Impact Assess Rev . (2023) 99:107035. doi: 10.1016/j.eiar.2023.107035

47. Zhao, XC, Jiang, M, Wu, ZJ, and Zhou, Y. Quantitative evaluation of China’s energy security policy under the background of intensifying geopolitical conflicts: based on PMC model. Resour Policy . (2023) 85:104032. doi: 10.1016/j.resourpol.2023.104032

48. Xiong, YC, Zhang, CL, and Qi, H. How effective is the fire safety education policy in China? A quantitative evaluation based on the PMC-index model. Saf Sci . (2023) 161:106070. doi: 10.1016/j.ssci.2023.106070

49. Yang, T, Xing, C, and Li, XY. Evaluation and analysis of new-energy vehicle industry policies in the context of technical innovation in China. J Clean Prod . (2021) 281:125126. doi: 10.1016/j.jclepro.2020.125126

50. Liu, Q, Jia, MT, and Xia, D. Dynamic evaluation of new energy vehicle policy based on text mining of PMC knowledge framework. J Clean Prod . (2023) 392:136237. doi: 10.1016/j.jclepro.2023.136237

51. Wei, Q, Wang, XY, Zhang, GR, Li, XG, Yang, XJ, and Gu, DX. Internet healthcare policy analysis, evaluation, and improvement path: multidimensional perspectives. Healthcare (Basel) . (2023) 11:1905. doi: 10.3390/healthcare11131905

52. Zhang, CH, Li, XW, Sun, YF, Chen, J, and Streimikiene, D. Policy modeling consistency analysis during energy crises: evidence from China’s coal power policy. Technol Forecast Soc Change . (2023) 197:122931. doi: 10.1016/j.techfore.2023.122931

53. Chen, TG, Rong, JT, Peng, LJ, Yang, JJ, Cong, GD, and Fang, J. Analysis of social effects on employment promotion policies for college graduates based on data Mining for Online use Review in China during the COVID-19 pandemic. Healthcare (Basel) . (2021) 9:846. doi: 10.3390/healthcare9070846

54. Li, SQ, Shen, JQ, Sun, FH, Jia, YZ, and Han, HK. Quantitative evaluation of ecological compensation policies for the watershed in China: based on the improved policy modeling consistency index. Environ Sci Pollut R . (2022) 29:66659–74. doi: 10.1007/s11356-022-20503-8

55. Li, ZH, and Guo, XS. Quantitative evaluation of China’s disaster relief policies: a PMC index model approach. Int J Disaster Risk Reduct . (2022) 74:102911. doi: 10.1016/j.ijdrr.2022.102911

56. Liu, LM, Chen, JF, Wang, CB, and Wang, Q. Quantitative evaluation of China’s basin ecological compensation policies based on the PMC index model. Environ Sci Pollut R . (2023) 30:17532–45. doi: 10.1007/s11356-022-23354-5

57. Zhang, YK, Wang, T, Wang, CB, and Cheng, CG. Quantitative evaluation of China’s CSR policies based on the PMC-index model. Sustain For . (2023) 15:7194. doi: 10.3390/su15097194

58. Liu, YW, Li, J, and Xu, Y. Quantitative evaluation of high-tech industry policies based on the PMC-index model: a case study of China’s Beijing-Tianjin-Hebei region. Sustain For . (2022) 14:9338. doi: 10.3390/su14159338

59. Roy, R. Industrial innovation and public policy preparing for the 1980s and the 1990s. Design . (1981) 13.

60. Xu, Y, Li, J, and Liu, YW. Quantitative evaluation of provincial government plans for public health emergencies in China. Int J Disaster Risk Reduct . (2022) 81:103292. doi: 10.1016/j.ijdrr.2022.103292

61. Jiang, ZZ, and Gao, XW. Text mining and quantitative evaluation of China’s green consumption policies based on green consumption objects. Environ Dev Sustain . (2023) 26:6601–6622. doi: 10.1007/s10668-023-02976-w

62. He, ZY, Chu, ZJ, Zhao, MY, Zhuang, J, and Liu, FR. Policy-making coordination of municipal solid waste policies in China: a content analysis. J Mater Cycles Waste Manag . (2018) 20:1073–84. doi: 10.1007/s10163-017-0668-3

63. Zhang, ZH, Cheng, SQ, Wang, C, Song, SF, and Feng, YC. Climate policy uncertainty and corporate investment efficiency: evidence from China. J Environ Plan Manag . (2023) 1–21. doi: 10.1080/09640568.2023.2276062

64. Dolowitz, DP, and Marsh, D. Learning from abroad: the role of policy transfer in contemporary policy-making. Governance . (2000) 13:5–24. doi: 10.1111/0952-1895.00121

65. O’Faircheallaigh, C. Public participation and environmental impact assessment: purposes, implications, and lessons for public policy making. Environ Impact Assess Rev . (2010) 30:19–27. doi: 10.1016/j.eiar.2009.05.001

66. Dong, F, and Liu, YJ. Policy evolution and effect evaluation of new-energy vehicle industry in China. Resour Policy . (2020) 67:101655. doi: 10.1016/j.resourpol.2020.101655

67. Kuang, B, Han, J, Lu, XH, Zhang, XP, and Fan, XY. Quantitative evaluation of China’s cultivated land protection policies based on the PMC-index model. Land Use Policy . (2020) 99:105062. doi: 10.1016/j.landusepol.2020.105062

68. Ghazinoory, S, and Aghaei, P. Differences between policy assessment & policy evaluation; a case study on supportive policies for knowledge-based firms. Technol Forecast Soc Change . (2021) 169:120801. doi: 10.1016/j.techfore.2021.120801

69. Yu, JK, Yin, W, and Liu, DH. Evolution of mariculture policies in China: experience and challenge. Mar Policy . (2020) 119:104062. doi: 10.1016/j.marpol.2020.104062

70. Liu, F, Liu, G, Wang, X, and Feng, Y. Whether the construction of digital government alleviate resource curse? Empirical evidence from Chinese cities. Resour Policy . (2024) 90:104811. doi: 10.1016/j.resourpol.2024.104811

71. Chu, ZJ, Wu, BY, He, ZY, Zhuang, J, and Wang, WN. The policy-making trend analysis of municipal solid waste in China 1980–2015. Waste Manag Res . (2019) 37:601–10. doi: 10.1177/0734242x19836722

72. Huang, N-C, Kuo, H-W, Hung, T-J, and Hu, SC. Do healthy City performance awards Lead to health in all policies? A case of Taiwan. Int J Environ Res Public Health . (2019) 16:1061. doi: 10.3390/ijerph16061061

Keywords: PMC-Index model, medicine innovation policy, China, quantitative evaluation, policy formulation

Citation: Guo D, Qi L and Song X (2024) Quantitative evaluation of the medicine innovation policy in China: based on the PMC-Index model. Front. Public Health . 12:1403320. doi: 10.3389/fpubh.2024.1403320

Received: 19 March 2024; Accepted: 01 May 2024; Published: 16 May 2024.

Reviewed by:

Copyright © 2024 Guo, Qi and Song. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Dan Guo, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Open access
• Published: 16 May 2024

Experiences of UK clinical scientists (Physical Sciences modality) with their regulator, the Health and Care Professions Council: results of a 2022 survey

• Mark McJury 1  

BMC Health Services Research volume  24 , Article number:  635 ( 2024 ) Cite this article

Metrics details

In healthcare, regulation of professions is an important tool to protect the public. With increasing regulation however, professions find themselves under increasing scrutiny. Recently there has also been considerable concern with regulator performance, with high profile reports pointing to cases of inefficiency and bias. Whilst reports have often focused on large staff groups, such as doctors, in the literature there is a dearth of data on the experiences of smaller professional groups such Clinical Scientists with their regulator, the Health and Care Professions Council.

This article reports the findings of a survey from Clinical Scientists (Physical Sciences modality) about their experiences with their regulator, and their perception of the quality and safety of that regulation.

Between July–October 2022, a survey was conducted via the Medical Physics and Engineering mail-base, open to all medical physicists & engineers. Questions covered typical topics of registration, communication, audit and fitness to practice. The questionnaire consisted of open and closed questions. Likert scoring, and thematic analysis were used to assess the quantitative and qualitative data.

Of 146 responses recorded, analysis was based on 143 respondents. Overall survey sentiment was significantly more negative than positive, in terms of regulator performance (negative responses 159; positive 106; significant at p  < 0.001). Continuous Professional Development audit was rated median 4; other topics were rated as neutral (fitness to practice, policies & procedures); and some as poor (value).

Conclusions

The Clinical Scientist (Physical Sciences) professional registrants rated the performance of their regulator more negatively than other reported assessments (by the Professional Standards Authority). Survey respondents suggested a variety of performance aspects, such as communication and fitness to practice, would benefit from improvement. Indications from this small dataset, suggest a larger survey of HCPC registrants would be useful.

Peer Review reports

In Healthcare, protection of patients and the public is a core principle. Part the framework of protections, includes regulation of professions [ 1 ]. This aims to mitigate risks such as the risk from bogus practitioners – insufficiently trained people acting as fully-trained professional practitioners, see Fig.  1 .

Recent UK media report on a bogus healthcare practitioner [ 2 ]

Regulation of professions ensures that titles (e.g. Doctor, Dentist, Clinical Scientist) are protected in law. The protected title means someone may only use that title, if they are on the national register, managed by the regulator – the Health and Care Professions Council (HCPC). It is a criminal offence to use a protected title if you are not entitled to do so [ 3 ]. There are a large number of regulators in healthcare – see Table  1 . Most of the regulators manage a register for one profession, except the HCPC which regulates 15 professions.

To be included on the register, a candidate must meet the regulators criteria for knowledge and training, and a key element to remain, is to show evidence of continuous professional development (CPD). Being on the register ensures that a practitioner has met the appropriate level of competence and professional practice.

For many healthcare workers, being on the HCPC register is a compulsory requirement to be appointable to a post. They must pay the necessary annual fees, and abide by the policies drawn-up by the regulator, and generally professions have no choice of regulator – these are statutory bodies, setup by government.

Recently, there has been considerable public dissatisfaction with the activity & performance of some regulators, notably Ofwat [ 4 ], and Ofgem [ 5 ]. Healthcare workers should expect a high level of professionalism, efficiency, and integrity from a regulator, as the regulator’s performance directly affects staff and public safety.

In terms of the regulation of UK Clinical Scientists, there is a dearth of data regarding experiences with the HCPC and views on the quality of regulation provided.

Findings are reported here from a 2022 survey of Medical Physicists and Engineers (one of the 16 job roles or ‘modalities’ under the umbrella of Clinical Scientist). The research aim was to assess experiences, and the level of ‘satisfaction’ with the regulator. For the remainder of this report, the term Clinical Scientist will be taken to mean Clinical Scientist (Medical Physicist/Engineer). The survey was designed to gather & explore data about opinions and experiences regarding several key aspects of how the HCPC performs its role, and perception of the quality & safety of regulation delivered.

A short survey questionnaire was developed, with questions aimed to cover the main regulatory processes, including registration & renewal, CPD audit, and fitness-to-practice. There were also questions relating more generally to HCPC’s performance as an organisation, e.g. handling of personal data. Finally, participants were asked to rate the HCPC’s overall performance and what they felt was the ‘value’ of regulation. The survey questions are listed in the Supplementary file along with this article.

Questions were carefully worded and there was a balance of open and closed questions. A five-point Likert score was used to rate closed questions. The survey was anonymous, and the questions were not compulsory, allowing the responders to skip irrelevant or difficult questions. The survey also aimed to be as short & concise as possible, to be a minimal burden to busy clinical staff & hopefully maximise response rate. There were a small number of questions at the start of the survey, to collect basic demographics on the respondents (role, grade, UK nation etc.).

The survey was advertised on the online JISC-hosted UK Medical Physics and Engineering (UKMPE) mail-base. This offered convenient access for the majority of Clinical Scientists. The survey was advertised twice, to allow for potential work absence, holiday/illness etc. It was active from the end of July 2002 until October 2022, when responses appeared to saturate.

The data is a combination of quantitative rating scores, and qualitative text responses. This allows a mixed-methods approach to data analysis, combining quantitative assessment of the Likert scoring, and (recursive) thematic analysis of the free-text answers [ 6 ]. Thematic analysis is a standard tool, and has been reported as a useful & appropriate for assessing experiences, thoughts, or behaviours in a dataset [ 7 ]. The survey questions addressed the main themes, but further themes were identified using an inductive, data-driven approach. Qualitative data analysis (QDA) was performed using NVivo (QSR International).

Two survey questions attempted to obtain an overall perception of HCPC’s performance: the direct one (Q12), and a further question’Would you recommend HCPC as a regulator…?’. This latter question doesn’t perhaps add anything more, and in fact a few respondents suggested it was a slightly awkward question, given professions do not have a choice of regulator – so that has been excluded from the analysis.

Study conduct was performed in accordance with relevant guidelines and regulations [ 8 , 9 ]. Before conducting the survey of Clinical Scientists, the survey was sent to their professional body, the Institute of Physics and Engineering in Medicine (IPEM). The IPEM Professional Standards Committee reviewed the survey questions [ 10 ]. Written informed consent was obtained from participants.

Data analysis

Data was collected via an MS form, in a single excel sheet and stored on a secure network drive. The respondents were anonymised, and the data checked for errors. The data was then imported into NVivo v12.

Qualitative data was manually coded for themes, and auto-coded for sentiment. An inductive approach was used to develop themes.

The sample size of responses allowed the use of simple parametric tests to establish the level of statistical significance.

Survey demographics

A total of 146 responses were collected. Two respondents noted that they worked as an HCPC Partner (a paid role). They were excluded from the analysis due to potential conflict of interest. One respondent’s responses were all blank aside from the demographic data, so they were also excluded from further analysis.

Analysis is based on 143 responses, which represents ~ 6% of the UK profession [ 11 ]. It is arguable whether it is representative of the profession at this proportion of response – but these responses do offer the only sizeable pool of data currently available. The survey was aimed at those who are on the statutory register as they are most likely to have relevant interactions & experiences of the HCPC, but a small number of responses were also received from Clinical Technologists (Medical Technical Officers-MTOs) and Engineers (CEs) and these have been included in the analysis. Figure  2 shows the breakdown in respondents, by nation.

Proportion of respondents, by nation

Of the respondents, 91% are registered Clinical Scientists, and would therefore have a broad range of experience with HCPC and its processes. Mean time on the register was 12 yrs. Respondents show a large range in seniority, and their roles are shown in Fig.  3 (CS-Clinical Scientist; CE-Clinical Engineer; MTO-Medical Technical Officer/Technician; CS-P are those working in private healthcare settings, so not on Agenda for Change (AfC) pay bands).

Breakdown in respondents, by role and pay banding

These data can be compared with the most recent HCPC ‘snapshot’ of the CS registrants (find here: Registrants by profession snapshot—1967 to 2019 | ( https://www.hcpc-uk.org/resources/data/2019/registrant-snapshot/ )).

The perception of overall regulator performance, can be assessed in two ways – one interview question directly asked for a rating score, and the overall survey sentiment also offers additional insight.

The score for overall performance was a median of 3 (mean 2.7; response rate 90%) which suggests neutral satisfaction.

Respondents were not asked directly to explain this overall performance rating – themes were extracted from the questionnaire as a whole.

The auto-coded sentiment scores generated in the NVivo software are shown in Table  2 . There is a significantly stronger negative sentiment than positive for HCPC performance – moderate, strong and total sentiment scores are all higher for negative sentiment. The normal test for a single proportion (109), shows the negative and positive sentiment differences have statistical significance with p  < 0.001. Whilst the PSA assessment of HCPC performance in 2022–23 shows 100% performance for 4 out of 5 assessment areas, survey data here from regulated professionals suggests considerably less satisfaction with HCPC. This raises associated questions about the relevance and validity of PSA assessment.

A large number of respondents seem to question the value of regulation. Whilst many accepted the value for it in terms of protecting the safety of the public, many questioned its relevance & benefit to themselves. Many respondents also queried the payment model where although the main beneficiaries of regulation are the public & the employer, it is the registrants actually pay the fees for registration. There was very little mention in survey responses, of benefit in terms of protected-title. These issues were amalgamated into Theme 1— Value of regulation , with the two sub-themes Value in monetary terms (value-for-money) and Value in professional terms (benefit and relevance to the individual professional) (see Table  3 ).

In the survey, several aspects of HCPC organisational performance were scored – handling of personal data, registration and renewal, engagement with the profession, audit, and the quality and usefulness of HCPC policies. These formed Theme 2 and its sub-themes.

A third theme Registrant competence and vulnerability , was developed to focus on responses to questions related to the assessment of registrant competence and Fitness To Practice (FTP) processes.

Finally, the survey also directly asked respondents if they could suggest improvements which would have resulted in higher scoring for regulation quality and performance. These were grouped into Theme 4.

Theme 1 – Value of regulation

Value in monetary terms.

The Likert score for value-for-money was a median of 2 (mean 2.3; response rate 100%) which suggests dissatisfaction. This is one of the few survey questions to elicit a 100% response rate – a clear signal of its importance for registrants.

There was a high number of responses suggesting fees are too expensive (and a significantly smaller number suggesting good value). This ties in with some respondents explaining that the ‘benefit’ from registration is mainly for the employer (an assurance of high quality, well-trained staff). Several respondents point to little ‘tangible’ benefit for registrants and query whether the payment model is fair and if the employer should pay registrant fees.

“Expensive fees for what appears to be very little support.” Resp094

“It seems that I pay about £100 per year to have my name written on a list. It is unclear to me what the HCPC actually does in order to justify such a high fee.” Resp014

“I get, quite literally, nothing from it. It’s essentially a tax on work.” Resp008

Several respondents suggested that as registration was mandated by the employer, it was in essence an additional ‘tax’ on their employment, which was highlighted previously by Unison [ 12 ]. A comparator for payment model, are the checks preformed on potential staff who will be working with children and vulnerable adults. In general, these ‘disclosure’ checks are paid for by the employer, however the checks are not recurrent cost for each individual, but done once at recruitment.

Value in professional terms & relevance

This was not a direct question on the questionnaire, but emerged consistently in survey responses. Aside from value-for-money, the value of regulation can also refer to more general benefit and relevance for a professional, for example in protecting a professional title or emphasising the importance of a role. Many respondents commented, in relation to the ‘value’ of regulation, about the relevance of the HCPC to them and their job/role.

The largest number of responses highlighted the lack of clarity about HCPC’s role, and also to note its lack of relevance felt by a significant proportion of respondents.

“Not sure I have seen any value in my registration except that it is a requirement for my role” Resp017

“I really fail to understand what (sic) the benefits of registration.” Resp018

“They do not promote the profession. I see no evidence of supporting the profession. I pay to have the title and I am not aware of any other benefits.” Resp038

Theme 2 – HCPC performance

Communication & handling data.

The survey questionnaire did not have a specific question relating to communication, therefore no specific Likert scores are available. Rather, communication was a sub-theme which emerged in survey responses. The response numbers related to positive (1) and negative experiences (50) clearly suggest an overall experience of poor communication processes (and statistically significant at p  < 0.001 for a normal proportion test).

One respondent noted they had ‘given up’ trying to communicate with HCPC electronically. Several respondents also noted issues with conventional communication—letters from HCPC going to old addresses, or being very slow to arrive.

“…I have given up on contacting by electronic means.” Resp134

When trying to renew their registration, communication with HCPC was so difficult that two respondents noted they raised a formal complaint.

A number of respondents noted that when they eventually got through to the HCPC, staff were helpful, so the main communication issue may relate to insufficiently resourced lines of communication (phones & email) or the need for a more focussed first point of contact e.g. some form of helpdesk or triaging system.

“Recently long wait to get through to speak to someone… Once through staff very helpful.” Resp126

This topic overlaps with the next (Processing Registration & renewals) in that both involve online logins, website use etc.

Security & data handling was rated as neutral (median 3, mean 3.4; response rate 91%). Although responses were balanced in terms of satisfaction, a significant number noted a lack of knowledge about HCPC processes. There are almost equal proportions of respondents reporting no issues, some problems with handling of personal data, or insufficient knowledge to express an opinion.

Registration and renewal

The score for processing registrations & renewals, was a median of 4 (mean 3.5; response rate 92%) which suggests modest satisfaction.

The overall rating also suggests that the issues may have been experienced by a comparative minority of registrants and that for most, renewal was straightforward.

“They expected people to call their phone number, which then wasn’t picked up. They didn’t reply to emails except after repeated attempts and finally having to resort to raising a complaint.” Resp023

“Difficult to get a timely response. Difficult to discuss my situation with a human being…” Resp044

Although the Likert score is positive, the themes in responses explaining the rating, are more mixed. Many respondents mentioned either having or knowing others who had issues with registration renewal, and its online processes including payments. A few respondents mentioned that the process was unforgiving of small errors. One respondent, for example, missed ticking a box on the renewal form, was removed from the register and experienced significant difficulties (poor communication with HCPC) getting the issue resolved.

Some respondents noted issues related to a long absence from work (e.g. maternity/illness etc.) causing them to miss registration deadlines – for some, this seems to have resulted in additional fees to renew registration. It seems rather easy for small errors (on either side) to result in registrants being removed from the register. For registrants, this can have very serious consequences and it can then be difficult and slow to resolve this, sometimes whilst on no pay. There have also been other reported instances of renewal payment collection errors [ 13 ].

“I had been off work… and had missed their renewal emails…I was told that there would be no allowances for this situation, and I would have to pay an additional fee to re-register…” Resp139.

Some respondents raised the issue of exclusion – certain staff groups not being included on the register—such as Clinical Technologists and Clinical Engineers. This desire for inclusion, also points to a perception of value in being on the register. One respondent raised an issue of very difficult and slow processing of registration for a candidate from outside the UK.

“Staff member who qualified as medical physicist abroad…has had a dreadful, drawn out and fruitless experience.” Resp135

Overall, many respondents noted difficulties in renewing registration and issues with HCPC’s online processes. Some of these issues (e.g. website renewal problems) may have been temporary and are now resolved, but others (e.g. available routes for registration) remain to be resolved.

Audit process & policies

In the survey, 12% respondents reported having been audited by HCPC regarding their CPD (response rate 97%). This is well above the level of 2.5% of each profession, which HCPC aims to review at each renewal [ 14 ], and similar values reported by some professional bodies [ 15 ]. The participants seem representative, although two respondents mentioned their perception of low audit rates. Data on CPD audit is available here: https://www.hcpc-uk.org/about-us/insights-and-data/cpd/cpd-audit-reports/

Respondents rated the process of being audited as a median of 4 (mean 3.7), which is the joint highest score on the survey, pointing to satisfaction with the process. From the responses, the overall perception could be summed up as straight-forward, but time-consuming. Without regular record-keeping, unfortunately most audits will be time-consuming – the HCPC more so, as it is not an annual audit, but covers the two preceding years.

Some respondents did find the process not only straight-forward, but also useful (related to feedback received). However, responses regarding feedback were mixed, with comments on both good, and poor feedback from HCPC.

“Not difficult but quite long-winded” Resp008

“Very stressful and time consuming” Resp081

“While it was a lot of work the process seemed very thorough and well explained.” Resp114

The HCPC’s policies & procedures were rated as a median of 3 (mean 3.2; response rate 98%). This neutral score could suggest a mixture of confidence in HCPC practise. This score may also reflect the fact that the majority of respondents had either not read, or felt they had no need to read the policies, and so are largely unfamiliar with them.

The reasons for this lack of familiarity are also explained by some respondents – four commented that the policies & procedures are rather too generic/vague. Three respondents noted that they felt the policies were not sufficiently relevant to their clinical roles to be useful. This may be due to the policies being written at a level to be applicable to registrants from all 16 modalities – and perhaps a limitation of the nature of HCPC as a very large regulator. Familiarity seemed mainly to be restricted to policies around registration, and CPD. There were slightly lower response levels for positive sentiment (6), than negative sentiment (9).

“I’ve never had cause to read them.” Resp115

“Detached from the real clinical interface for our professions…” Resp083

HCPC split their policies into ‘corporate’- which relate to organisational issues (e.g. equality & diversity; find them here: Our policies and procedures | ( https://www.hcpc-uk.org/about-us/corporate-governance/freedom-of-information/policies/#:~:text=Our%20main%20policies%20and%20procedures%201%20Customer%20feedback,scheme%20...%207%20Freedom%20of%20Information%20Policy%20 )) and those more relevant to professions (e.g. relating to the register; find them here: Resources | ( https://www.hcpc-uk.org/resources/?Query=&Categories=76 )).

One respondent noted not only that the policies were ‘as you might expect’, but felt the policies were less demanding than those from other similar bodies such as the CQC ( https://www.cqc.org.uk/publications ).

“…Other regulatory bodies (such as the CQC for example) have policies and procedures that are a lot more challenging to comply with.” Resp022

Theme 3 – Registrant competence and vulnerability

In this survey, 3.5% (5/143) of respondents noted some involvement with the HCPC’s Fitness to Practice service. These interactions were rated at a median of 3 (mean 2.8) suggesting neutral sentiment.

Firstly, we can immediately see the level of interaction with the FTP team is very small. CS registrants represent approx. 2% of HCPC registrants, and the level of CS referrals to FTP in 2020–21 was 0.2% [ 16 ].

The data is a very small sample, but responses vary strongly, so it is worth digging a little further into the granularity of individual responses. Response scores were 1, 1, 2, 5, 5 – which are mainly at the extremes of the rating spectrum. The majority of respondents described poor experiences with the FTP team: errors, a process which was ‘extremely prolonged’, involved slow/poor communication, and processes which were ‘entirely opaque’.

“It is slow, the process was badly managed… and the system was entirely opaque,” Resp37

“They were hard to contact and I didn't feel they listened…no explanation, apology or assurance it would not happen again. It left my colleague disillusioned and me very angry on their behalf…” Resp044

Some respondents commented that the team were not only difficult to contact, but also didn’t seem to listen. At the end of a process which involved errors from HCPC, one respondent noted were ‘no explanation, apologies or assurance that it would not happen again’, leaving the registrant ‘disillusioned’. These experiences do not fit with the HCPC’s stated goal to be a compassionate regulator, see Fig.  4 . Arguably it is more difficult to change a culture of behaviour and beliefs, than to publish a corporate goal or statement of vision.

HCPC’s vision statement & purpose [ 17 ]

Some survey respondents have noted the necessity of regulation for our profession.

“Ultimately I am very grateful that I can register as a professional.” Resp024

Theme 4 – Suggestions for improved regulation

Following the question relating to overall performance, respondents were invited to suggest things which might improve their rating for HCPC’s performance and value. These suggestions were also combined with those which appeared in earlier survey responses.

Although we are in a current cost-of-living crisis, responses did not query simply high absolute cost of fees, but also queried the value/benefit of HCPC regulation for registrants. Many responses expressed doubt as to the added value & relevance of HCPC registration for them. They seem to point to a desire for more tangible benefit from their fees. Perhaps, given the costs and levels of scrutiny, registrants want some definite benefit to balance the scales .

“Cost less and do more for the people who are on the register.” Resp089

“Vastly reduced cost. Employer paying registrant fees.” Resp074

A significant number of responses pointed out that the main benefits of registration are for the public, and for employers – but that it is the registrants who pay for registration. Many queries why this should be, and whether there should be a different payment model, where for example employers pay.

Similarly, some respondents felt that the HCPC’s unusual position of regulating a large swathe of healthcare professions was not necessarily helpful for their profession or others.

Communication and response times are obviously an issue of concern for registrants, and improvements are needed based on the low satisfaction levels reported here. This is also linked to a wish for increased engagement with the CS profession.

“Engagement with the workforce, specialism specific development, reduced fees” Resp025

Some responses suggested they would be comforted by increased accountability / governance of HCPC including improved FTP efficiency.

“More accountability to registrants” Resp130

Finally, improvement in terms of additional registration routes for Engineers & Technical staff were also suggested. It may be damaging to work-place moral, if two professionals doing roles of a similar nature are not being governanced is the same way and if there is not parity of their gross salary due to mandatory professional fees & reductions.

Value-for-money : This will vary between individuals depending on many variables, such as upbringing & environment, salary, lifestyle priorities, political persuasion, and so on. However, many of these factors should balance in a large sample. In general, it can be suggestive of satisfaction (or lack of) with a service. The score here suggesting dissatisfaction, echoes with other reports on HCPC’s spending, and financial irregularities [ 18 , 19 ].

In the survey findings, respondents have voiced dissatisfaction with registration value for money. In fact, HCPC’s registration fees are not high when compared to the other healthcare professions regulators. Table 1 shows data from 2021–22 for regulator annual registration fees. However, the HCPC has risen from having the lowest regulator fees in 2014–5, to its current position (9 th of 13) slightly higher in the table. Perhaps more concerning than the absolute level of fees, are when large increases are proposed [ 12 , 20 , 21 , 22 ].

However, fees have regularly increased to current figure of £196.48 for a two-year cycle. During a consultation process in 2018, the Academy for Healthcare Clinical Scientists (AHCS) wrote an open letter to the HCPC, disputing what they felt was a disproportionate fee increase [ 23 ]. Further fee rises have also been well above the level of inflation at the time.

HCPC expenditure (which is linked to registration fees) has arguably been even more controversial than fee increases – noted by several respondents. A freedom of information (FOI) request in 2016 showed HCPC’s spending of £17,000 for their Christmas party [ 18 ] – which amounts to just over £76 per person. This cost was close to the annual registration fee (at that time) for registrants.

In 2019, regulation of social workers in England moved from HCPC, to Social Work England. This resulted in a loss of over 100,000 registrants, and a loss in registration fee income. HCPC raised fees to compensate, but a freedom of information (FoI) request in 2020 [ 18 ] showed that even though there was an associated lowering in workload associated with the loss of 100 k registrants, the HCPC had no redundancies, suggesting the loss of income was compensated mainly by the fees increase.

Inherent value & relevance

One of HCPC’s aims is to promote ‘the value of regulation’ [ 24 ]. However, not only is there dissatisfaction with value-for-money, the second highest response suggests a lack of inherent value (or benefit) from regulation to the individual registrant. In some ways, there is a lack of balance – registrants are under increasing scrutiny, but feel there is little direct benefit, to provide balance.

This also suggests that HCPC’s aim or message is not getting through to the CS profession. It’s not clear what the HCPC 2021–22 achieved milestone – ‘Embedded our registrant experiences research into employee learning and development and inductions’ has actually achieved.

A large number of responses pointed to the lack of clarity about HCPC’s role, and also to note its lack of relevance for respondents. Some of this is understandable – until recently, many CS registrants will have little interaction with HCPC. They would typically get one email reminder each year to renew their registration and pay those fees, and hear little else from the HCPC. That is beginning to change, and HCPC have recently begun to send more regular, direct emails/updates to registrants.

However, for many registrants, the HCPC appears not to be clearly communicating its role, or the relevance/importance of regulation. As mentioned above, this also links in to previous mentions of the lack of any tangible benefit for registrants. Some note little more relevance other than the mandatory aspects of regulation.

Finally, relevance is also queried in relation to the limited access for some professional groups to a professional register. The current situation of gaps in registration for some groups, results in two situations – firstly, for Clinical Scientists and Clinical Engineers/Technologists, one group has to compulsorily pay a fee to be allowed/approved to do their job and the other does not; also, the public are routinely helped and assisted by Clinical Scientists and Clinical Engineers/Technologists – but only one group is regulated to ensure public safety.

HCPC Communication

This was highlighted by respondents as often poor. Recently in the media, there has been a concern raised by The College of Paramedics (CoP) about communications issues with HCPC—changes to the HCPC policy on the use of social media [ 25 ]. They raised particular concerns about the use of social media content and ‘historical content’ in the context of investigations of fitness-to practice.

There have previously been some concerns raised on the UKMPE mail-base regarding handling of personal data, and lack of efficiency in addressing the issue [ 26 ]. Several messages detailed HCPC communicating unencrypted registrant passwords in emails and sending personal data to the incorrect registrant. Some on the forum noted that they had reported this problem over a period of several years to HCPC, suggesting HCPC’s response to these serious issues was extremely slow. Several responses noted these previous issues.

Registration processes

Although responses here show some satisfaction, there have been reports in the media of significant issues with registration (such as removing registrants from the register in error) with associated impact for patients and the public [ 27 , 28 ]. Similarly, there were reports on the UKMPE mail-base of significant issues with registration renewals being problematic [ 26 ]. In Scotland, NHS.net email accounts ceased to be supported in July-Sept 2020 and the associated lack of access to email accounts and messages used for HCPC communication and registration, caused a major issue in registration renewal. This coincided with COVID lockdowns and a period of unusually difficult communication with HCPC. If NHS staff lose registration (irrespective of the reason), respondents noted that some Human Resources (HR) departments were quick to suspend staff from work, and in some cases withhold pay. That spike in difficulties is likely the cause of the most common responses suggesting issues with a complicated process.

In safe-guarding public safety, a key task for a healthcare regulator is assessing the competence of registrants. This is done via a small set of related activities. Registrants must return regular evidence of CPD, and these are audited for 2.5% registrants. This process is simple and routine, and as seen in Theme 2 responses here suggest registrants are reasonably satisfied with this process.

More formal and in-depth competence assessment happens when a complaint is raised against a registrant, either by a work colleague/management, a member of the public or occasionally by the HCPC itself. The process is complex, lengthy and can end in a registrant attending a court hearing [ 29 ].

It is usual for registrants to continue in their normal job during FTP investigations – effectively the public remains at risk from a registrant if their competence is eventually proven to be below the regulators standards, so there is a need for investigations to be efficient both in timeliness, and outcome.

Obviously, being under investigation can be highly stressful, and has the potential for the registrant to be ‘struck off’ the register, and lose their job if registration is mandated (e.g. NHS posts). There are many reports of the process & experience either provoking or increasing underlying mental health challenges [ 30 , 31 , 32 ]. Along with efficiency, a regulator needs to behave compassionately. Investigations of highly-skilled professionals engaging in complex work activities, is also necessarily complex and requires a high degree of knowledge and experience from the regulator’s investigational panel.

The Professional Standards Authority (PSA) regulate the HCPC, and publish annual reviews of their performance ( https://www.professionalstandards.org.uk/publications/performance-reviews ) (see Table  4 ). HCPC performance as reported by PSA, seems to be generally higher than noted by survey respondents here. For 2022–23, aside from one area, the HCPC has scored 100% for performance, which seems at odds with these survey responses [ 33 ]. The FTP team is notable in repeatedly performing very poorly compared to most other sections of the HCPC (even though the majority of the HCPC budget goes to FTP activity, see Fig.  4 ). The HCPC Annual Report 2018–9 [ 34 ] highlighted the completion of the first phase of the Fitness-To-Practice Improvement Plan. This delivered “A root and branch review of this regulatory function… a restructure, tightened roles and processes and the introduction of a new Threshold Policy”, but this seems to have no impact on the performance reported by the PSA for the next few years shown in Table  4 . However, the most recent data does suggest improvement, and HCPC continues to develop FTP team practice [ 17 ].

HCPC expenditure for the year 2020–21 [ 17 ]

There are other reports of poor experiences with this team [ 35 , 36 ], and in one report the FTP team’s processes have been noted as being rather inhumane [ 35 ].

Regulation is an important part of public protection, but how effectively it is managed & enforced is also a concern, given it involves increased scrutiny of registrants. A topical comparator is the current dissatisfaction by a large section of the public about several other government regulators allowing seemingly poor performance to go unchecked [ 4 , 5 ].

It is arguable, that registrants remain on the register as long as the HCPC allows them. Several respondents in this survey noted being removed from the register through HCPC administrative error. Removal could also happen through poor judgement/decision-making – the FTP team handle large numbers of very complex investigational cases – 1603 concluded cases for the year 2021–22 and 1024 hearings [ 16 ]. Every justice system is subject to a level of error – guilty parties can be erroneously ‘cleared’, and vice-versa. It is essential therefore, that policies & procedures relating to FTP are fit for purpose—that the FTP team work effectively and humanely, and that there is genuine & effective governance of HCPC to ensure accountability. In this survey, some respondents seem to be saying that currently this seems not to be the case.

It might have been anticipated that the greatest concern is costs, especially in the current cost-of-living crisis. The recent HCPC consultation to increase fees [ 37 ] seems particularly tone-deaf and has caused concern across the professions [ 21 , 22 ].

Above findings show respondents are interested in lower fees, but also increased benefit for their fees. Some respondents pointed out that whilst registrants pay for registration, benefit is mainly for the public and employers. The HCPC is a statutory body, its funding model will have been designed/decided upon by government, and may be unlikely to change. However, there are a variety of potential regulation models [ 38 ], and so change is possible. A review of the financial model for regulation may be welcome.

Regulator size

Some aspects of HCPC performance, policies, and distribution of spending, is related to the nature of it being the largest and only multi-professional regulator in the healthcare sector. Data from the HCPC suggests (see Fig.  5 ) that the majority of spending relates to FTP activity. Data also points to Clinical Scientists having very low levels of FTP investigation compared to others in HCPC [ 16 ]. This suggests that a significant proportion of CS registrant fees are used to investigate other professions. It’s possible (perhaps simplistically) that if, like many other healthcare professions such as doctors & dentists who’s regulator is concerned only with that single profession, if CSs were regulated separately, their registrant fees may be reduced. This model of single-profession regulation may also mitigate against other disadvantages of the HCPC’s practice, such as the ‘generic’ policies aiming to apply to a pool of 15 professions.

Although there is a very low level of data for this topic, the concerned raised by registrants are serious in nature. There also seems to be issues in handling of complaints related to this service and advocacy for registrants. Certainly, there is a clear governance path via PSA, to the Health Secretary. However, this does not offer a route for individual complaints to be raised and addressed. Unlike complaints from the public in other areas, there is no recourse to an ombudsman for registrants. The only option for individual registrants, is the submission of a formal complaint to the HCPC itself, which is dealt with internally. Comments from survey respondents suggest this process does not guarantee satisfaction. Indeed, one of the respondents who mentioned submitting a complaint, made it clear they remained unhappy with HCPC’s response. Overall, there seems to be a lack of clear & effective advocacy for registrants.

“…the HCPC’s stance appeared to be guilty until proven innocent… At no point did I feel the HCPC cared that their (sic) was an individual involved....” Resp044.

FTP processes affect a comparatively small number of CS registrants, compared to other professions. However, it seems clear that the majority of those who have interacted with the FTP team have had poor experiences, and respondents have suggested improvements are needed. The reason for FTP investigations, is protection of staff and the public. If processes are slow, and investigations prolonged, or decisions flawed, the public may be exposed to increased levels of risk, as healthcare practitioners who may be lacking in competence continue to practice. The data in Table  4 shows concerning but improving trends in FTP performance levels.

Limitations

There are two main limitations to this work. Firstly, due to time constraints, there was no pilot work done when designing the survey questionnaire. This may have helped, as noted earlier, a few responses pointed to some awkwardness with one survey question. Although no pilot work was done, the questionnaire was reviewed by the IPEM Professional Standards Committee, as noted in the Acknowledgements section.

The other obvious limitation is the low response rate (~ 6% of UK Medical Physicists). Circulation of the survey was performed via the only online forum for the profession currently available. The survey was advertised multiple times to ensure visibility to staff who may have missed it initially due to leave etc. However, the forum does reach 100% of the profession, and some addressees may have filters set to send specific posts to junk folders etc. The professional body IPEM declined to offer support in circulating the survey (believing the issues involved would affect/be of interest only to a small minority of members.)

The low response rate also has a particular impact on the pool of responses relating to FTP issues, which inherently affect low numbers of registrants.

However, the importance of some of the findings here (e.g. expressed dissatisfaction with regulation in terms of value; the poor experience of some members with the Registration, Communication and FTP teams) and the low sample surveyed, both justify the need for a larger follow-on survey, across all of Clinical Science.

In Healthcare, regulation of professions is a key aspect of protecting the public. However, to be effective, regulation must be performed professionally, impartially, and associated concerns or complaints investigated efficiently and respectfully.

This report presents findings from a survey aimed at collecting a snap-shot of the experiences of Clinical Scientists with their regulator, and their perception of the quality and safety of that regulation performance.

Overall survey sentiment scores showed a significantly more negative responses than positive. Survey comments relate not only to current issues, but to previous problems and controversial issues [ 18 , 26 ]. It seems that some respondents have at some point lost confidence and trust in the HCPC, and survey responses suggest there has not been enough engagement and work done by HCPC to repair and rebuild this trust.

In the midst of a cost of living crisis, costs are a large concern for many. The HCPC fees are neither the highest not lowest amongst the healthcare regulators. Spending is transparent, and details can be found in any of the HCPC’s annual reports.

A repeating sub-theme in responses, was a lack of tangible value for the registrant, and that the employer should pay the costs of registration, where registration is mandated by the job.

Many respondents have suggested that they feel there should be more proactive engagement from HCPC with the profession. Most respondents were not familiar with or felt the HCPC policies are relevant/important to them.

Survey data showed moderate satisfaction with registration processes for the majority of respondents. Some respondents also noted a lack of registration route for engineering & technical healthcare staff. CPD processes also achieved a score indicating registrant satisfaction. This generated the highest ratings in the survey. Communication scored poorly and many respondents suggests there needs to be improved levels of communication in terms of response times and access to support.

The CS profession experiences low levels of interaction with the FTP service. However, those interactions which were recorded in the survey, show some poor experiences for registrants. There also seems to be a lack of advocacy/route for complaints about HCPC from individual registrants. There may need to be more engagement between registrants and their professional body regarding HCPC performance, and more proactivity from the stake-holder, IPEM.

Some of the findings reported here relate to important issues, but the survey data are based on a low response rate. A larger survey across all of Clinical Science is being planned.

Availability of data and materials

To protect confidentiality of survey respondents, the source data is not available publicly, but are available from the author on reasonable request.

Abbreviations

Agenda for Change

Academy for Healthcare Clinical Scientists

Continuous professional development

Clinical Engineer

Clinical Scientist

College of Paramedics

Clinical Technologist

Freedom of Information

Fitness-to-practice

Health and Care Professions Council

Human resources

Institute of Physics and Engineering in Medicine

Joint Information Systems Committee

Medical Technical Officer

Professional Standards Authority

Professional Standards Committee

Qualitative data analysis

UK Medical Physics and Engineering

Professional Standards Authority. Professional healthcare regulation in the UK. https://www.professionalstandards.org.uk/news-and-blog/blog/detail/blog/2018/04/10/professional-healthcare-regulation-explained#:~:text=Regulation%20is%20simply%20a%20way,may%20face%20when%20receiving%20treatment . Accessed 26 Jul 2023

Evening Standard. Bogus surgeon treated hundreds. https://www.standard.co.uk/hp/front/bogus-surgeon-treated-hundreds-6326549.html . Accessed 26 Jul 2023.

HCPC . About registration: protected titles. http://www.hcpc-uk.org/aboutregistration/protectedtitles/ . Accessed 27 Jul 23.

The Guardian. Public patience is wearing thin. Ofwat must wield the big stick | Nils Pratley |  https://www.theguardian.com/business/nils-pratley-on-finance/2022/dec/08/public-patience-is-wearing-thin-ofwat-must-wield-the-big-stick . Accessed 19 Jul 2023.

TrustPilot. Reviews of Ofgem. Ofgem Reviews | Read Customer Service Reviews of ofgem.com (trustpilot.com). Accessed 19 Jul 2023.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Article   Google Scholar  

Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE Guide No. 131. Med Teach. 2020;42(8):846–54.

Article   PubMed   Google Scholar  

Declaration of Helsinki. 2013. https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/ . Accessed 12 Sept 2023.

UK Data Protection Act. 2018. https://www.gov.uk/data-protection . Accessed 15 Sept 2023.

Rowbottom C. Private communication on behalf of the IPEM Professional Standards Committee; 2022.

IPEM Workforce Team. Clinical scientist & engineer workforce data. Personal communication. 2022.

Unison. HCPC fee increase is an unjustified ‘tax on practising.’ https://www.unison.org.uk/news/press-release/2019/02/hcpc-fee-increase-unjustified-tax-practising/ . Accessed 27 Jul 2023.

HCPC. Direct debit collection errors. https://www.hcpc-uk.org/news-and-events/news/2020/early-direct-debit-collections/?dm_i=2NJF,141CO,7C0ZNI,4A8IE,1 . Accessed 27 Jul 23.

HCPC. CPD audit rates. https://www.hcpc-uk.org/cpd/cpd-audits/ . Accessed 21 Jul 2023.

IPEM. CPD audit rates. https://www.ipem.ac.uk/your-career/cpd-career-development/cpd-audit/ . Accessed 21 Jul 2023.

HCPC. Fitness to practice annual report 2020–21. https://www.hcpc-uk.org/about-us/insights-and-data/ftp/fitness-to-practise-annual-report-2020-21/ . Accessed 23 Jul 2023.

HCPC. Annual report and accounts, 2020–21. https://www.hcpc-uk.org/resources/reports/2022/annual-report-and-accounts-2020-21/ . Accessed 19 Jul 2023.

Wikipedia. The health and care professions council. https://en.wikipedia.org/wiki/Health_and_Care_Professions_Council . Accessed 2 Jul 23.

HCPC. Annual report 2005–06. https://www.hcpc-uk.org/resources/reports/2006/annual-report-2005-06/ . Accessed 19 Jul 2023.

British Dental Association. BDA very disappointed by HCPC decision to raise registration fees by 18%. https://www.bda.uk.com/resource/bda-very-disappointed-by-hcpc-decision-to-raise-registration-fees-by-18.html . Accessed 27 Jul 2023.

British Psychological Society. HCPC fees consultation – share your views. https://www.bps.org.uk/news/hcpc-fee-consultation-share-your-views . Accessed 27 Jul 23.

IBMS. IBMS response to the HCPC registration fees consultation. https://www.ibms.org/resources/news/ibms-response-to-hcpc-registration-fees-consultation/ . Accessed 17 Jul 23.

Association of HealthCare Scientists. Open letter to HCPC. https://www.ahcs.ac.uk/wp-content/uploads/2018/11/HCPC-Open-Letter.pdf . Accessed 27 Jul 23.

HCPC. Corporate plan 2022–23. https://www.hcpc-uk.org/resources/reports/2022/hcpc-corporate-plan-2022-23/ . Accessed 23 Jul 2023.

College of Paramedics. Our formal response to the HCPC consultation. https://collegeofparamedics.co.uk/COP/News/2023/Our%20formal%20response%20to%20the%20HCPC%20consultation.aspx . Accessed 27 Jul 23.

JISC Mail - MPE mailbase. JISCMail - Medical-physics-engineering list at www.jiscmail.ac.uk . Accessed 19 July 2023.

The Guardian. Thousands miss out on treatment as physiotherapists are taken off UK register. https://www.theguardian.com/society/2022/may/14/thousands-miss-out-on-treatment-as-physiotherapists-are-struck-off-uk-register . Accessed 27 Jul 2023.

HSJJobs.com. https://www.hsjjobs.com/article/thousands-of-clinicians-unable-to-work-after-registration-blunder . Accessed 27 Jul 2023.

HCPC. How we investigate. https://www.hcpc-uk.org/concerns/how-we-investigate/ . Accessed 21 Nov 2023.

Sirriyeh R, Lawton R, Gardner P, Armitage G. Coping with medical error: a systematic review of papers to assess the effects of involvement in medical errors on healthcare professionals’ psychological well-being. Br Med J Qual Saf. 2010;19:6.

Google Scholar  

Bourne T, Wynants L, Peters M, van Audenhove C, Timmerman D, van Calster B, et al. The impact of complaints procedures on the welfare, health and clinical practise of 7926 doctors in the UK: a cross-sectional survey. BMJ Open. 2015;5:e006687.

Article   PubMed   PubMed Central   Google Scholar  

Jones-Berry S. Suicide risk for nurses during fitness to practice process. Ment Health Pract. 2016;19:8.

Professional Standards Authority. HCPC performance review 2022–23. https://www.professionalstandards.org.uk/publications/performance-review-detail/periodic-review-hcpc-2022-23 . Accessed 25 Jul 2023

HCPC. Annual report and accounts, 2018–19. https://www.hcpc-uk.org/resources/reports/2019/hcpc-annual-report-and-accounts-2018-19/ . Accessed 19 Jul 2023.

Maben J, Hoinville L, Querstret D, Taylor C, Zasada M, Abrams R. Living life in limbo: experiences of healthcare professionals during the HCPC fitness to practice investigation process in the UK. BMC Health Serv Res. 2021;21:839–54.

Leigh J, Worsley A, Richard C, McLaughlin K. An analysis of HCPC fitness to practise hearings: fit to practise or fit for purpose? Ethics Soc Welfare. 2017;11(4):382–96.

HCPC. Consultation changes to fees. https://www.hcpc-uk.org/news-and-events/consultations/2022/consultation-on-changes-to-fees/ . Accessed 27 Jul 23

Department of Health. Review of the regulation of public health professions. London: DoH; 2010.

Download references

Acknowledgements

The author wishes to kindly acknowledge the input of Dr Carl Rowbottom (IPEM Professional Standards Committee), in reviewing the survey questions. Thanks also to Dr Nina Cockton for helpful advice on ethics and recruitment issues.

There were no sources of funding required for this work.

Author information

Authors and affiliations.

University of Glasgow, Level 2, ICE Building, Queen Elizabeth University Hospital Campus, 1345 Govan Road, Glasgow, G51 4TF, UK

Mark McJury

You can also search for this author in PubMed   Google Scholar

Contributions

All work to collect, analyse & publish this survey, are the work of the author Dr Mark McJury.

Corresponding author

Correspondence to Mark McJury .

Ethics declarations

Ethics approval and consent to participate.

As this study relates to low risk, survey data, formal ethics committee approval is not required (exemption obtained from NHSGGC REC04 REC Officer Dr Judith Godden [email protected]). As the survey responses were from members of a professional body (The Institute of Medical Physics and Engineering in Medicine (IPEM) it was consulted. Its Professional Standards Committee (PSC) reviewed the survey and raised no objections. The survey questions were assessed for bias and approved unchanged (acknowledged in the manuscript). Written informed consent was obtained from all participants in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

The survey questionnaire has been provided as a supplementary file.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

McJury, M. Experiences of UK clinical scientists (Physical Sciences modality) with their regulator, the Health and Care Professions Council: results of a 2022 survey. BMC Health Serv Res 24 , 635 (2024). https://doi.org/10.1186/s12913-024-10956-7

Download citation

Received : 06 September 2023

Accepted : 05 April 2024

Published : 16 May 2024

DOI : https://doi.org/10.1186/s12913-024-10956-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Regulation of professions
• Clinical scientists
• Medical physicists

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Quantitative determination of residue amounts of pesticide active ingredients used in grapes by LC-MS/MS and GC-MS/MS devices and evaluation of these pesticides in terms of public health

• Research Article
• Published: 17 May 2024

Cite this article

• Ayhan Elmastas   ORCID: orcid.org/0000-0002-9208-9852 1  

The aim of this study was to quantitatively determine pesticide residues in grapes, one of the most produced and consumed fruits in Turkey and in the world. A total of 226 active ingredients were analyzed in 21 samples collected from Southeastern and Eastern Anatolia regions using QuEChERS (quick, easy, cheap, effective, rugged, and safe) extraction method and multiple residue analysis technique and LC-MS/MS and GC-MS/MS devices. In 11 out of 21 samples (52.4%), no active ingredient was detected, while at least one active ingredient was detected in 10 samples (47.6%). Thirteen different active substances (Ametoctradin, Azoxystrobin, Boscalid, Diphenoconazole, Dimethomorph, Fenhexamid, Fluopyram, Flutriafol, Metalaxyl- Metalaxyl-M, Metrafenone, Tebuconazole, Trifloxystrobin) were detected in the samples. The top 3 most detected active substances were Boscalid-Azoxystrobin and Fluopyram, respectively. The active ingredients were found between 0.015 and 0.499 mg kğ −1 values.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Data availability

The corresponding author will provide the data that support the study’s conclusions upon reasonable request.

Alonso Gonzalez P, Parga Dans E, Acosta Dacal AC, Pena MZ, Luzardo OP (2022) Differences in the levels of sulphites and pesticide residues in soils and wines and under organic and conventional production methods. J Food Compos Anal 112:104714. https://doi.org/10.1016/j.jfca.2022.104714

Article   CAS   Google Scholar  

Anonymous, 2021. Tarım ve Orman Bakanlığı. Türk Gıda Kodeksi Pestisitlerin Maksimum Kalıntı Limitleri Yönetmenliği Resmi Gazete: 27.09.2021 Sayı: 31611

Attaullah M, Yousuf MJ, Shaukat S, Anjum SI, Ansari MJ, Buneri ID, Tahir M, Amin M, Ahmad N, Khan SU (2018) Serum organochlorine pesticides residues and risk of cancer: a case-control study. Saudi Journal of Biological Sciences 25(7):1284–1290. https://doi.org/10.1016/j.sjbs.2017.10.023

Bajwa U, Sandhu KS (2014) Effect of handling and processing on pesticide residues in food– a review. J Food Sci Technol 51(2):201–220. https://doi.org/10.1007/s13197-011-0499-5

Baldi I, Gruber A, Rondeau V, Lebailly P, Brochard P, Fabrigoule C (2011) Neurobe-havioral effects of long-term exposure to pesticides: results from the 4-year follow-up of the phytoner study. Occup Environ Med 68(2):108–115. https://doi.org/10.1136/oem.2009.047811

Cesnik HB, Gregorcic A, Cus F (2008) Pesticide residues in grapes from vineyards included in integrated pest management in Slovenia. Food Addit Contam 25(4):438–443. https://doi.org/10.1080/02652030701558490

Codex Alimentarius Comission, 1984. Guide to Codex recommendations concerning pesticide residues. Part 1.General notes and guidelines. The Hague. Codex Alimentarius Comission Report CAC/PRI-1984

Datapool, 2022. https://www.eurl-pesticides-datapool.eu (Accessed July 17 2022).

EFSA 2022 https://www.efsa.europa.eu/en/topics/topic/pesticides (Accessed May 24 2022).

Elmastas A, Umaz A, Aydın F (2021) Investigation of changes against time in the artificial gastric liquid medium of pesticide active substances. Journal of the Institute of Science and Technology 11(2):1270–1278. https://doi.org/10.21597/jist.772406

Article   Google Scholar  

Elmastas A, Umaz A, Pirinç V, Aydın F (2022) Evaluation as time-dependent of pesticides applied in pre-harvest period of grown vegetables: removal of pesticide residues in the vegetables. Int J Environ Anal Chem. https://doi.org/10.1080/03067319.2022.2085040

Elmastas A, Umaz A, Pirinç V, Aydın F (2023) Quantitative determination and removal of pesticide residues in fresh vegetables and fruit products by LC–MS/ MS and GC–MS/MS. Environ Monit Assess 195:277. https://doi.org/10.1007/s10661-022-10910-2

European Commission, 1984. Pesticides-advice and recommendations to be used by national and other authorities as well as manufactures concerned with the registration of agricultural and non agricultural pesticides 6 th ed. Strasbourg 120 pp.

European Commission (2002) Commission Directive 2002/63/EC of 11 July 2002 establishing community methods of sampling for the official control of pesticide residues in and on products of plant and animal origin and repealing Directive 79/700/EEC. Off J Eur Communities 2:30–43

Google Scholar  

European Commission,2019. Analytıcal qualıty control and method valıdatıon procedures for pestıcıde resıdues analysıs ın food and feed SANTE/12682/ 2019 Eur Comm Dir Heal Food Saf.

FAO (1986) International code of conduct on the distribution and use of pesticides. Food and Agriculture Organization of the United Nations, Rome, p 28

Fenik J, Tankiewicz M, Biziuk M (2011) Properties and determination of pesticides in fruits and vegetables. TrAC Trends Anal Chem 30(6):814–826. https://doi.org/10.1016/j.trac.2011.02.008

Kim KH, Kabir E, Jahan SA (2017) Exposure to pesticides and the associated human health effects. Sci Total Environ 575:525–535. https://doi.org/10.1016/j.scitotenv.2016.09.009

Lehotay SJ (2007) Determination of pesticide residues in foods by acetonitrile extraction and partitioning with magnesium sulfate: collaborative study. J AOAC Int 90(2):485–520

Otteneder H, Majerus P (2005) Pesticide residues in wine, transfer from grapes. Bulletin OIV 78(889-890):173–181

CAS   Google Scholar  

Sharma A, Shukla A, Attri K, Kumar M, Kumar P, Suttee A, Singh G, Barnwal R, Singla N (2020) Global trends in pesticides: a looming threat and viable alternatives. Ecotoxicol Environ Saf 201:110812. https://doi.org/10.1016/j.ecoenv.2020.110812

Slavin JL, Lloyd B (2012) Health benefits of fruits and vegetables. Adv Nutr 3(4):506–516. https://doi.org/10.3945/an.112.002154

Song NE, Lee JY, Mansur AR, Jang HW, Lim M-C, Lee Y, Yoo M, Nam TG (2019) Determination of 60 pesticides in hen eggs using the QuEChERS procedure followed by LC-MS/MS and GC-MS/MS. Food Chem 298:125050. https://doi.org/10.1016/j.foodchem.2019.125050

Van Maele-Fabry G, Lantin AC, Hoet P, Lison D (2010) Childhood leukemia and parental occupational exposure to pesticides: a systematic review and meta-analysis. Cancer Causes Control 21:787–809. https://doi.org/10.1007/s10552-010-9516-7

Walorczyk S, Drozdzyski D, Kowalska J, Remlein-Starosta D, Ziolkowski A, Przewozniak M, Gnusowski B (2013) Pesticide residues determination in Polish organic crops in 2007-2010 applying gas chromatography-tandem quadrupole mass spectrometry. Food Chem 139:482–487. https://doi.org/10.1016/j.foodchem.2013.01.013

WHO (1990). Puplıc Health Impact Of Pesticides used in agrıgulture ISBN 92 4 156139 4 World Health Organization Geneva 1990

WHO (2012). GEMS/Food cluster diets 2012. Data-Consumption-2012. http://www.who.int/foodsafety/chem/Cluster_diets_2012_consumption.xls .

Wu ML, Wu YC, Chen YC (2019) Detection of pesticide residues on intact tomatoes by carbon fiber ionization mass spectrometry. Anal Bioanal Chem 411:1095–1105. https://doi.org/10.1007/s00216-018-1539-z

Yakar Y (2018) Çekirdeksiz Sofralık Üzümlerde Pestisit Kalıntılarının Belirlenmesi. YYÜ Tar Bil Derg 28(4):444–447. https://doi.org/10.29133/yyutbd.453960

Zhao P, Huang B, Li Y, Han Y, Zou N, Gu K, Li X, Pan C (2014) Rapid multiplug filtration cleanup with multiple-walled carbon nanotubes and gas chromatography-triple-quadrupole mass spectrometry detection for 186 pesticide residues in tomato and tomato products. J Agric Food Chem 62(17):3710–3725. https://doi.org/10.1021/jf405240j

Download references

Author information

Authors and affiliations.

Ministry of Agriculture and Forestry, Gap International Agricultural Research and Training Center Directorate, 21100, Diyarbakır, Turkey

Ayhan Elmastas

You can also search for this author in PubMed   Google Scholar

Contributions

Ayhan ELMASTAŞ: Pesticide investigation, data collection, methodology, formal analysis, discussed the results, commented on the manuscript and the original draft writing

Corresponding author

Correspondence to Ayhan Elmastas .

Ethics declarations

Ethics approval.

The author declares that this study includes no experiments with human or animal subjects. Ethics approval is not required for this study

Consent to participate

This manuscript does not contain any individual person’s data in any form. There is no consent needed for this manuscript.

Consent for publication

The author has read and approved the paper for publication.

Competing interests

The author declares no competing interests.

Additional information

Responsible Editor: Ester Heath

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Elmastas, . Quantitative determination of residue amounts of pesticide active ingredients used in grapes by LC-MS/MS and GC-MS/MS devices and evaluation of these pesticides in terms of public health. Environ Sci Pollut Res (2024). https://doi.org/10.1007/s11356-024-33693-0

Download citation

Received : 27 March 2024

Accepted : 12 May 2024

Published : 17 May 2024

DOI : https://doi.org/10.1007/s11356-024-33693-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Pesticide residue
• LC-MS-MS -GC-MS-MS
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sage Choice

The value of qualitative methods to public health research, policy and practice

Institute of Mental Health, University of Nottingham Faculty of Medicine and Health Sciences, Triumph Road, Nottingham NG7 2UH, UK

A O’Caithain

The University of Sheffield, Sheffield, UK

Sheffield Hallam University, Sheffield, UK

This article reviews the role and use of qualitative methods in public health research.

‘Signs of quality’ are introduced to help guide potential authors to publish their qualitative research in public health journals. We conclude that high-quality qualitative research offers insights that quantitative research cannot. It is time for all public health journals to recognise the value of qualitative research and increase the amount that they publish.

Introduction

In this article, we briefly review the role and use of qualitative methods in public health research and its significance for research, policy and practice. Historically, public health research has been largely dependent on quantitative research rooted in medical science. Qualitative research approaches, however, are able to provide the ‘lived experience’ perspective of patients, practitioners and the public on any aspect of public health.

To inform this article, we searched the most recent original research articles published in ten of the most widely cited public health journals in the world (generally those with the highest impact factor, including Perspectives in Public Health ). The list of journals can be found in Table 1 .

The methods used in 100 recently published original research articles in 10 public health journals

We examined 100 of the most recently published original research articles (10 from each journal up until May, 2021) to discover how many of these reported qualitative methods. The findings from this quick review can be found in Table 1 below. The review revealed that 85 articles reported quantitative methods, 11 reported mixed-methods, and only 4 reported qualitative methods. In our review, we deliberately did not include one public health journal, Critical Public Health because it specialises in publishing qualitative public health research studies. With only four qualitative research papers out of the most recent 100 public health original research articles published in the top journals, we have decided to publish this article first to encourage qualitative research practices in public health, second to highlight the value of qualitative research, third to briefly identify what makes ‘good qualitative research’ and finally to promote increased submissions of original qualitative research in this and other public health journals.

Reporting Qualitative Health Research

Qualitative research has its origins in Interpretivism. As such, it has been widely used in the social sciences, in contrast to the medical sciences that historically have largely embraced the positivist tradition. Typically, public health research has followed the positivist tradition although qualitative research methodology appears more often in public health journals than top medical journals. For example, a cursory examination of the Lancet indicates that it does not appear to publish any qualitative research and the British Medical Journal ( BMJ ) rarely does so. In 2016, the BMJ published an open letter from 76 senior academics from 11 countries inviting its editors to: ‘ … reconsider their policy of rejecting qualitative research on the grounds of low priority. They challenge the journal to develop a proactive, scholarly, and pluralist approach to research that aligns with its stated mission ’. 1 Included in their support for qualitative research articles in the BMJ , they observe that many of the journal’s top papers have been qualitative studies. This letter has been cited 250 times in the literature, largely supportive of their views. In their reply to the letter, Editors of the BMJ acknowledge that: ‘ … we agree they can be valuable, and recognise that some research questions can only be answered by using qualitative methods ’. 2 In so much as we can tell to date, the BMJ has not changed its practice. Fortunately, published accounts of qualitative research in various other health disciplines flourishes, for example, there are now at least two health journals that are exclusively designed for this purpose ( Qualitative Health Research and International Journal of Qualitative Studies on Health and Well-being ).

The Value of Qualitative Health Research

The following quotation succinctly argues the need for qualitative research methods in public health:

Public health, we believe, needs both epidemiology and qualitative research. Without epidemiology we cannot answer questions about the prevalence of and association between health determinants and outcomes. Without qualitative enquiry, it is difficult to explain how individuals interpret health and illness in their everyday lives, or to understand the complex workings of the social, cultural and institutional systems that are central to our health and wellbeing. 3

In particular, given a situation with complex phenomena involving human experience and behaviour, quantitative research may equally excel in finding out ‘what and when?’, but qualitative research may equally be needed to find out ‘why, how and how come?’. Green and Britten 4 summarise the role of qualitative research in health, and we have adapted their key points to apply to public health:

• Qualitative methods can help bridge the gap between scientific evidence and public health policy and practice by investigating human perceptions and experiences.
• Recognising the limits of quantitative approaches and that different research questions require different kinds of research.
• Qualitative research findings provide rigorous and firsthand accounts of public health educational, promotional and clinical practices in everyday contexts.
• Qualitative research can be used to help inform individual health choices and health promotion initiatives within communities.

Doing High-Quality Qualitative Research

Quality is unlikely to be the only reason that so little qualitative research finds its way into public health journals; even research articles of the highest quality may be met with resistance from reviewers and editors. Nonetheless it is important to attend to quality. Articles using qualitative methods require the same rigour as articles reporting quantitative methods; however, the criteria for assessing rigour are different. When assessing qualitative articles, we need to remember that what is considered rigorous in the social sciences is not necessarily the same as what is considered rigorous in the medical sciences and vice versa. Either way, what is important is that public health journals publish high-quality research studies, whatever methodology is employed. The following quotation is helpful in focusing on the need for rigour in qualitative approaches to healthcare research:

The use of qualitative research in health care enables researchers to answer questions that may not be easily answered by quantitative methods. Moreover, it seeks to understand the phenomenon under study in the context of the culture or the setting in which it has been studied … (however this) … requires researchers in health care who attempt to use it, to have a thorough understanding of its theoretical basis, methodology and evaluation techniques. 5

As quoted above, Al-Busaidi, 5 asserts that qualitative health researchers need an appreciation of theory and methodologies and use of both in all research and evaluation studies. What is most important in any qualitative study is that the research question is clear and the method is appropriate to answer the research question. We can therefore begin to ask critical questions of any qualitative article submitted for publication in public health journals:

• Is the research question clear?
• Is the method appropriate for addressing the research question?
• Is there an explanation as to how and why this method is appropriate?
• What are the theories referred to in this study and how are these applied?
• Are these theories consistent throughout the study?
• Has the sample been critiqued to make readers aware of who is not included and how this might affect findings?
• Is the analysis grounded in the data?
• Does the analysis address questions of the data so that insights are identified that go beyond simply describing what participants have said?
• Are there clearly articulated implications for public health practice?

In addition to these fundamental questions, to help researchers report qualitative research, there are two frameworks that help to maintain standards for the conduct and reporting of the method. The first is COREQ (Consolidated criteria for reporting qualitative research). 6 This is a 32-point checklist of three domains: research team and reflexivity, study design and analysis and findings. The second is Standards for Reporting Qualitative Research (SRQR), 7 which is a 21-point check-list following the same format. Together, these are both useful tools for helping researchers think about what they need to consider when conducting qualitative research and for helping reviewers assess articles using qualitative methods. We are not suggesting that qualitative researchers should use these frameworks as tick-box checklists, although they may be used to enable researchers to think through important elements of qualitative research that may be otherwise overlooked. At the end of this article, we supply weblinks to enable the reader to inspect these two frameworks.

‘Signs of Quality’ for Reporting Qualitative Public Health Research

Rather than leave the reader baffled by frameworks and checklists, we propose a number of ‘signs of quality’ that we would expect to see when reviewing articles submitted to this or any other high-quality public health journal.

The research question is clearly identified and clearly related to public health policy or practice and the chosen method is appropriate for answering that question. A rationale is offered to justify the study and the methods used.

Ethical questions are considered, the study has been conducted and reported in an ethical manner, and ethical approval has been granted from a recognised ethics committee.

How the study was implemented needs to be reported as clearly as possible including: how access to participants was achieved, what questions were asked, and how the analysis was conducted.

The study needs to be both theoretically and practically consistent. For example, if the study claims to be narrative research, did the questions elicit stories and is narrative theory used in analysis?

Collaborative

In recent years, health services in many countries have embraced patient and public involvement and co-production in both research and practice. Such initiatives are designed to draw our attention to service users’ views, needs and desires. This agenda sits very well with qualitative research methodologies.

Contribution

Every research study needs to make a contribution to the body of knowledge concerning the subject under investigation. If there is theoretical and practical consistency throughout the study and it has been competently conducted and analysed, the reader should come away with a sense of learning something new on the topic. This insight should be easy for a reader to take away from each article and the easiest way to do this is to articulate it clearly in the conclusion in the abstract as well as the conclusion in the body of the paper. Conclusions of ‘it’s complex’ or ‘there were five issues affecting this phenomenon’ fail to offer useful insights. They may be a signal of an under-analysed study. It will be much more helpful to readers to state a single key issue that adds to the evidence base and that helps members of the population, policy-makers, or practitioners to understand the phenomenon under study or take action on it.

Examples of Good Qualitative Research from this Journal

In order to exemplify the principles, we espouse in this article, we refer to two recent articles published in Perspectives in Public Health that use qualitative methods. First, Lozano-Sufrategui et al. 8 aimed to ‘… understand the behaviour changes men who attended a weight loss programme engage in during weight maintenance … ’. To achieve this aim, the research team encouraged men on a weight loss programme to keep photo-diaries of themselves and to talk about their progress with the researchers. The research is innovative in its approach and uniquely reports the participants’ thoughts, feelings and behaviours. It highlights the importance of drawing on the diversity of methods that exist beyond face-to-face interviews. The second example is Eley et al. 9 who conducted interviews and focus groups in four countries in order to ‘ … explore school educators’ attitudes, behaviours and knowledge towards food hygiene, safety and education .’ Using this approach, they were able to explore individual and group views on this subject thus identifying not only the need for more educational resources but barriers and opportunities in the process. While reading these articles, it becomes immediately apparent that these studies were able to gain insight into the respective topics that quantitative methods could never achieve. What qualitative research facilitates is the human connection between interviewer and interviewee and in that process, together with the guarantee of confidentiality, people are able to speak in-depth about their experiences and perceptions, from which much can be learned. In these two examples, the qualitative findings give insights into the thoughts and feelings of the participants and enable a greater understanding of how the researchers were able to draw their conclusions from the research.

A review of top public health journals identified that the vast majority of research that is being currently published in high-ranking public health journals use quantitative methods. High-quality qualitative research offers insights that quantitative research cannot. It is time for all public health journals to recognise the value of qualitative research and increase the amount of high-quality qualitative research that they publish.

COREQ link :

http://cdn.elsevier.com/promis_misc/ISSM_COREQ_Checklist.pdf

SRQR link :

https://onlinelibrary.wiley.com/pb-assets/assets/15532712/SRQR_Checklist-1529502683197.pdf

Conflict of Interest: The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding: The author(s) received no financial support for the research, authorship and/or publication of this article.

Contributor Information

T Stickley, Institute of Mental Health, University of Nottingham Faculty of Medicine and Health Sciences, Triumph Road, Nottingham NG7 2UH, UK.

A O’Caithain, The University of Sheffield, Sheffield, UK.

C Homer, Sheffield Hallam University, Sheffield, UK.

Important Dates

Important Dates for 2024:

• Application posted online: November 1, 2023
• Application due date: CLOSED
• Notification of acceptances: March 2024
• Program start: June 10th, 2024
• Program end: July 19th, 2024