empirical research journals

Psychological Science

Prospective submitters of manuscripts are encouraged to read Editor-in-Chief Simine Vazire’s editorial , as well as the editorial by Tom Hardwicke, Senior Editor for Statistics, Transparency, & Rigor, and Simine Vazire.

Psychological Science , the flagship journal of the Association for Psychological Science, is the leading peer-reviewed journal publishing empirical research spanning the entire spectrum of the science of psychology. The journal publishes high quality research articles of general interest and on important topics spanning the entire spectrum of the science of psychology. Replication studies are welcome and evaluated on the same criteria as novel studies. Articles are published in OnlineFirst before they are assigned to an issue. This journal is a member of the Committee on Publication Ethics (COPE) .

Quick Facts

Read the February 2022 editorial by former Editor-in-Chief Patricia Bauer, “Psychological Science Stepping Up a Level.”

Read the January 2020 editorial by former Editor Patricia Bauer on her vision for the future of  Psychological Science .

Read the December 2015 editorial on replication by former Editor Steve Lindsay, as well as his April 2017 editorial on sharing data and materials during the review process.

Watch Geoff Cumming’s video workshop on the new statistics.

Current Issue

Online First Articles

List of Issues

Editorial Board

Submission Guidelines

Editorial Policies

Featured research from psychological science, talking politics with strangers isn’t as awful as you’d expect, research suggests.

Many of us avoid discussing politics with someone who holds an opposing viewpoint, assuming the exchange will turn nasty or awkward. But having those conversations is far more gratifying than we expect, new research suggests. 

Discerning Discoveries

New research shows why people often read more into a scientific finding than what the data might show.

Driving Simulation and AI Deepen Insights into Impulsivity 

Lab experiments sometimes have participants engage in tasks that don’t capture the full range of behaviors people display in their day-to-day lives, but pairing realistic tasks with machine learning could help researchers more accurately assess individuals’ personality traits.

Privacy Overview

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 01 June 2023

Data, measurement and empirical methods in the science of science

• Lu Liu 1 , 2 , 3 , 4 ,
• Benjamin F. Jones   ORCID: orcid.org/0000-0001-9697-9388 1 , 2 , 3 , 5 , 6 ,
• Brian Uzzi   ORCID: orcid.org/0000-0001-6855-2854 1 , 2 , 3 &
• Dashun Wang   ORCID: orcid.org/0000-0002-7054-2206 1 , 2 , 3 , 7  

Nature Human Behaviour volume  7 ,  pages 1046–1058 ( 2023 ) Cite this article

16k Accesses

8 Citations

119 Altmetric

Metrics details

• Scientific community

The advent of large-scale datasets that trace the workings of science has encouraged researchers from many different disciplinary backgrounds to turn scientific methods into science itself, cultivating a rapidly expanding ‘science of science’. This Review considers this growing, multidisciplinary literature through the lens of data, measurement and empirical methods. We discuss the purposes, strengths and limitations of major empirical approaches, seeking to increase understanding of the field’s diverse methodologies and expand researchers’ toolkits. Overall, new empirical developments provide enormous capacity to test traditional beliefs and conceptual frameworks about science, discover factors associated with scientific productivity, predict scientific outcomes and design policies that facilitate scientific progress.

Similar content being viewed by others

SciSciNet: A large-scale open data lake for the science of science research

Zihang Lin, Yian Yin, … Dashun Wang

Envisioning a “science diplomacy 2.0”: on data, global challenges, and multi-layered networks

Simone Turchetti & Roberto Lalli

Interdisciplinarity revisited: evidence for research impact and dynamism

Keisuke Okamura

Scientific advances are a key input to rising standards of living, health and the capacity of society to confront grand challenges, from climate change to the COVID-19 pandemic 1 , 2 , 3 . A deeper understanding of how science works and where innovation occurs can help us to more effectively design science policy and science institutions, better inform scientists’ own research choices, and create and capture enormous value for science and humanity. Building on these key premises, recent years have witnessed substantial development in the ‘science of science’ 4 , 5 , 6 , 7 , 8 , 9 , which uses large-scale datasets and diverse computational toolkits to unearth fundamental patterns behind scientific production and use.

The idea of turning scientific methods into science itself is long-standing. Since the mid-20th century, researchers from different disciplines have asked central questions about the nature of scientific progress and the practice, organization and impact of scientific research. Building on these rich historical roots, the field of the science of science draws upon many disciplines, ranging from information science to the social, physical and biological sciences to computer science, engineering and design. The science of science closely relates to several strands and communities of research, including metascience, scientometrics, the economics of science, research on research, science and technology studies, the sociology of science, metaknowledge and quantitative science studies 5 . There are noticeable differences between some of these communities, mostly around their historical origins and the initial disciplinary composition of researchers forming these communities. For example, metascience has its origins in the clinical sciences and psychology, and focuses on rigour, transparency, reproducibility and other open science-related practices and topics. The scientometrics community, born in library and information sciences, places a particular emphasis on developing robust and responsible measures and indicators for science. Science and technology studies engage the history of science and technology, the philosophy of science, and the interplay between science, technology and society. The science of science, which has its origins in physics, computer science and sociology, takes a data-driven approach and emphasizes questions on how science works. Each of these communities has made fundamental contributions to understanding science. While they differ in their origins, these differences pale in comparison to the overarching, common interest in understanding the practice of science and its societal impact.

Three major developments have encouraged rapid advances in the science of science. The first is in data 9 : modern databases include millions of research articles, grant proposals, patents and more. This windfall of data traces scientific activity in remarkable detail and at scale. The second development is in measurement: scholars have used data to develop many new measures of scientific activities and examine theories that have long been viewed as important but difficult to quantify. The third development is in empirical methods: thanks to parallel advances in data science, network science, artificial intelligence and econometrics, researchers can study relationships, make predictions and assess science policy in powerful new ways. Together, new data, measurements and methods have revealed fundamental new insights about the inner workings of science and scientific progress itself.

With multiple approaches, however, comes a key challenge. As researchers adhere to norms respected within their disciplines, their methods vary, with results often published in venues with non-overlapping readership, fragmenting research along disciplinary boundaries. This fragmentation challenges researchers’ ability to appreciate and understand the value of work outside of their own discipline, much less to build directly on it for further investigations.

Recognizing these challenges and the rapidly developing nature of the field, this paper reviews the empirical approaches that are prevalent in this literature. We aim to provide readers with an up-to-date understanding of the available datasets, measurement constructs and empirical methodologies, as well as the value and limitations of each. Owing to space constraints, this Review does not cover the full technical details of each method, referring readers to related guides to learn more. Instead, we will emphasize why a researcher might favour one method over another, depending on the research question.

Beyond a positive understanding of science, a key goal of the science of science is to inform science policy. While this Review mainly focuses on empirical approaches, with its core audience being researchers in the field, the studies reviewed are also germane to key policy questions. For example, what is the appropriate scale of scientific investment, in what directions and through what institutions 10 , 11 ? Are public investments in science aligned with public interests 12 ? What conditions produce novel or high-impact science 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ? How do the reward systems of science influence the rate and direction of progress 13 , 21 , 22 , 23 , 24 , and what governs scientific reproducibility 25 , 26 , 27 ? How do contributions evolve over a scientific career 28 , 29 , 30 , 31 , 32 , and how may diversity among scientists advance scientific progress 33 , 34 , 35 , among other questions relevant to science policy 36 , 37 .

Overall, this review aims to facilitate entry to science of science research, expand researcher toolkits and illustrate how diverse research approaches contribute to our collective understanding of science. Section 2 reviews datasets and data linkages. Section 3 reviews major measurement constructs in the science of science. Section 4 considers a range of empirical methods, focusing on one study to illustrate each method and briefly summarizing related examples and applications. Section 5 concludes with an outlook for the science of science.

Historically, data on scientific activities were difficult to collect and were available in limited quantities. Gathering data could involve manually tallying statistics from publications 38 , 39 , interviewing scientists 16 , 40 , or assembling historical anecdotes and biographies 13 , 41 . Analyses were typically limited to a specific domain or group of scientists. Today, massive datasets on scientific production and use are at researchers’ fingertips 42 , 43 , 44 . Armed with big data and advanced algorithms, researchers can now probe questions previously not amenable to quantification and with enormous increases in scope and scale, as detailed below.

Publication datasets cover papers from nearly all scientific disciplines, enabling analyses of both general and domain-specific patterns. Commonly used datasets include the Web of Science (WoS), PubMed, CrossRef, ORCID, OpenCitations, Dimensions and OpenAlex. Datasets incorporating papers’ text (CORE) 45 , 46 , 47 , data entities (DataCite) 48 , 49 and peer review reports (Publons) 33 , 50 , 51 have also become available. These datasets further enable novel measurement, for example, representations of a paper’s content 52 , 53 , novelty 15 , 54 and interdisciplinarity 55 .

Notably, databases today capture more diverse aspects of science beyond publications, offering a richer and more encompassing view of research contexts and of researchers themselves (Fig. 1 ). For example, some datasets trace research funding to the specific publications these investments support 56 , 57 , allowing high-scale studies of the impact of funding on productivity and the return on public investment. Datasets incorporating job placements 58 , 59 , curriculum vitae 21 , 59 and scientific prizes 23 offer rich quantitative evidence on the social structure of science. Combining publication profiles with mentorship genealogies 60 , 61 , dissertations 34 and course syllabi 62 , 63 provides insights on mentoring and cultivating talent.

This figure presents commonly used data types in science of science research, information contained in each data type and examples of data sources. Datasets in the science of science research have not only grown in scale but have also expanded beyond publications to integrate upstream funding investments and downstream applications that extend beyond science itself.

Finally, today’s scope of data extends beyond science to broader aspects of society. Altmetrics 64 captures news media and social media mentions of scientific articles. Other databases incorporate marketplace uses of science, including through patents 10 , pharmaceutical clinical trials and drug approvals 65 , 66 . Policy documents 67 , 68 help us to understand the role of science in the halls of government 69 and policy making 12 , 68 .

While datasets of the modern scientific enterprise have grown exponentially, they are not without limitations. As is often the case for data-driven research, drawing conclusions from specific data sources requires scrutiny and care. Datasets are typically based on published work, which may favour easy-to-publish topics over important ones (the streetlight effect) 70 , 71 . The publication of negative results is also rare (the file drawer problem) 72 , 73 . Meanwhile, English language publications account for over 90% of articles in major data sources, with limited coverage of non-English journals 74 . Publication datasets may also reflect biases in data collection across research institutions or demographic groups. Despite the open science movement, many datasets require paid subscriptions, which can create inequality in data access. Creating more open datasets for the science of science, such as OpenAlex, may not only improve the robustness and replicability of empirical claims but also increase entry to the field.

As today’s datasets become larger in scale and continue to integrate new dimensions, they offer opportunities to unveil the inner workings and external impacts of science in new ways. They can enable researchers to reach beyond previous limitations while conducting original studies of new and long-standing questions about the sciences.

Measurement

Here we discuss prominent measurement approaches in the science of science, including their purposes and limitations.

Modern publication databases typically include data on which articles and authors cite other papers and scientists. These citation linkages have been used to engage core conceptual ideas in scientific research. Here we consider two common measures based on citation information: citation counts and knowledge flows.

First, citation counts are commonly used indicators of impact. The term ‘indicator’ implies that it only approximates the concept of interest. A citation count is defined as how many times a document is cited by subsequent documents and can proxy for the importance of research papers 75 , 76 as well as patented inventions 77 , 78 , 79 . Rather than treating each citation equally, measures may further weight the importance of each citation, for example by using the citation network structure to produce centrality 80 , PageRank 81 , 82 or Eigenfactor indicators 83 , 84 .

Citation-based indicators have also faced criticism 84 , 85 . Citation indicators necessarily oversimplify the construct of impact, often ignoring heterogeneity in the meaning and use of a particular reference, the variations in citation practices across fields and institutional contexts, and the potential for reputation and power structures in science to influence citation behaviour 86 , 87 . Researchers have started to understand more nuanced citation behaviours ranging from negative citations 86 to citation context 47 , 88 , 89 . Understanding what a citation actually measures matters in interpreting and applying many research findings in the science of science. Evaluations relying on citation-based indicators rather than expert judgements raise questions regarding misuse 90 , 91 , 92 . Given the importance of developing indicators that can reliably quantify and evaluate science, the scientometrics community has been working to provide guidance for responsible citation practices and assessment 85 .

Second, scientists use citations to trace knowledge flows. Each citation in a paper is a link to specific previous work from which we can proxy how new discoveries draw upon existing ideas 76 , 93 and how knowledge flows between fields of science 94 , 95 , research institutions 96 , regions and nations 97 , 98 , 99 , and individuals 81 . Combinations of citation linkages can also approximate novelty 15 , disruptiveness 17 , 100 and interdisciplinarity 55 , 95 , 101 , 102 . A rapidly expanding body of work further examines citations to scientific articles from other domains (for example, patents, clinical drug trials and policy documents) to understand the applied value of science 10 , 12 , 65 , 66 , 103 , 104 , 105 .

Individuals

Analysing individual careers allows researchers to answer questions such as: How do we quantify individual scientific productivity? What is a typical career lifecycle? How are resources and credits allocated across individuals and careers? A scholar’s career can be examined through the papers they publish 30 , 31 , 106 , 107 , 108 , with attention to career progression and mobility, publication counts and citation impact, as well as grant funding 24 , 109 , 110 and prizes 111 , 112 , 113 ,

Studies of individual impact focus on output, typically approximated by the number of papers a researcher publishes and citation indicators. A popular measure for individual impact is the h -index 114 , which takes both volume and per-paper impact into consideration. Specifically, a scientist is assigned the largest value h such that they have h papers that were each cited at least h times. Later studies build on the idea of the h -index and propose variants to address limitations 115 , these variants ranging from emphasizing highly cited papers in a career 116 , to field differences 117 and normalizations 118 , to the relative contribution of an individual in collaborative works 119 .

To study dynamics in output over the lifecycle, individuals can be studied according to age, career age or the sequence of publications. A long-standing literature has investigated the relationship between age and the likelihood of outstanding achievement 28 , 106 , 111 , 120 , 121 . Recent studies further decouple the relationship between age, publication volume and per-paper citation, and measure the likelihood of producing highly cited papers in the sequence of works one produces 30 , 31 .

As simple as it sounds, representing careers using publication records is difficult. Collecting the full publication list of a researcher is the foundation to study individuals yet remains a key challenge, requiring name disambiguation techniques to match specific works to specific researchers. Although algorithms are increasingly capable at identifying millions of career profiles 122 , they vary in accuracy and robustness. ORCID can help to alleviate the problem by offering researchers the opportunity to create, maintain and update individual profiles themselves, and it goes beyond publications to collect broader outputs and activities 123 . A second challenge is survivorship bias. Empirical studies tend to focus on careers that are long enough to afford statistical analyses, which limits the applicability of the findings to scientific careers as a whole. A third challenge is the breadth of scientists’ activities, where focusing on publications ignores other important contributions such as mentorship and teaching, service (for example, refereeing papers, reviewing grant proposals and editing journals) or leadership within their organizations. Although researchers have begun exploring these dimensions by linking individual publication profiles with genealogical databases 61 , 124 , dissertations 34 , grants 109 , curriculum vitae 21 and acknowledgements 125 , scientific careers beyond publication records remain under-studied 126 , 127 . Lastly, citation-based indicators only serve as an approximation of individual performance with similar limitations as discussed above. The scientific community has called for more appropriate practices 85 , 128 , ranging from incorporating expert assessment of research contributions to broadening the measures of impact beyond publications.

Over many decades, science has exhibited a substantial and steady shift away from solo authorship towards coauthorship, especially among highly cited works 18 , 129 , 130 . In light of this shift, a research field, the science of team science 131 , 132 , has emerged to study the mechanisms that facilitate or hinder the effectiveness of teams. Team size can be proxied by the number of coauthors on a paper, which has been shown to predict distinctive types of advance: whereas larger teams tend to develop ideas, smaller teams tend to disrupt current ways of thinking 17 . Team characteristics can be inferred from coauthors’ backgrounds 133 , 134 , 135 , allowing quantification of a team’s diversity in terms of field, age, gender or ethnicity. Collaboration networks based on coauthorship 130 , 136 , 137 , 138 , 139 offer nuanced network-based indicators to understand individual and institutional collaborations.

However, there are limitations to using coauthorship alone to study teams 132 . First, coauthorship can obscure individual roles 140 , 141 , 142 , which has prompted institutional responses to help to allocate credit, including authorship order and individual contribution statements 56 , 143 . Second, coauthorship does not reflect the complex dynamics and interactions between team members that are often instrumental for team success 53 , 144 . Third, collaborative contributions can extend beyond coauthorship in publications to include members of a research laboratory 145 or co-principal investigators (co-PIs) on a grant 146 . Initiatives such as CRediT may help to address some of these issues by recording detailed roles for each contributor 147 .

Institutions

Research institutions, such as departments, universities, national laboratories and firms, encompass wider groups of researchers and their corresponding outputs. Institutional membership can be inferred from affiliations listed on publications or patents 148 , 149 , and the output of an institution can be aggregated over all its affiliated researchers 150 . Institutional research information systems (CRIS) contain more comprehensive research outputs and activities from employees.

Some research questions consider the institution as a whole, investigating the returns to research and development investment 104 , inequality of resource allocation 22 and the flow of scientists 21 , 148 , 149 . Other questions focus on institutional structures as sources of research productivity by looking into the role of peer effects 125 , 151 , 152 , 153 , how institutional policies impact research outcomes 154 , 155 and whether interdisciplinary efforts foster innovation 55 . Institution-oriented measurement faces similar limitations as with analyses of individuals and teams, including name disambiguation for a given institution and the limited capacity of formal publication records to characterize the full range of relevant institutional outcomes. It is also unclear how to allocate credit among multiple institutions associated with a paper. Moreover, relevant institutional employees extend beyond publishing researchers: interns, technicians and administrators all contribute to research endeavours 130 .

In sum, measurements allow researchers to quantify scientific production and use across numerous dimensions, but they also raise questions of construct validity: Does the proposed metric really reflect what we want to measure? Testing the construct’s validity is important, as is understanding a construct’s limits. Where possible, using alternative measurement approaches, or qualitative methods such as interviews and surveys, can improve measurement accuracy and the robustness of findings.

Empirical methods

In this section, we review two broad categories of empirical approaches (Table 1 ), each with distinctive goals: (1) to discover, estimate and predict empirical regularities; and (2) to identify causal mechanisms. For each method, we give a concrete example to help to explain how the method works, summarize related work for interested readers, and discuss contributions and limitations.

Descriptive and predictive approaches

Empirical regularities and generalizable facts.

The discovery of empirical regularities in science has had a key role in driving conceptual developments and the directions of future research. By observing empirical patterns at scale, researchers unveil central facts that shape science and present core features that theories of scientific progress and practice must explain. For example, consider citation distributions. de Solla Price first proposed that citation distributions are fat-tailed 39 , indicating that a few papers have extremely high citations while most papers have relatively few or even no citations at all. de Solla Price proposed that citation distribution was a power law, while researchers have since refined this view to show that the distribution appears log-normal, a nearly universal regularity across time and fields 156 , 157 . The fat-tailed nature of citation distributions and its universality across the sciences has in turn sparked substantial theoretical work that seeks to explain this key empirical regularity 20 , 156 , 158 , 159 .

Empirical regularities are often surprising and can contest previous beliefs of how science works. For example, it has been shown that the age distribution of great achievements peaks in middle age across a wide range of fields 107 , 121 , 160 , rejecting the common belief that young scientists typically drive breakthroughs in science. A closer look at the individual careers also indicates that productivity patterns vary widely across individuals 29 . Further, a scholar’s highest-impact papers come at a remarkably constant rate across the sequence of their work 30 , 31 .

The discovery of empirical regularities has had important roles in shaping beliefs about the nature of science 10 , 45 , 161 , 162 , sources of breakthrough ideas 15 , 163 , 164 , 165 , scientific careers 21 , 29 , 126 , 127 , the network structure of ideas and scientists 23 , 98 , 136 , 137 , 138 , 139 , 166 , gender inequality 57 , 108 , 126 , 135 , 143 , 167 , 168 , and many other areas of interest to scientists and science institutions 22 , 47 , 86 , 97 , 102 , 105 , 134 , 169 , 170 , 171 . At the same time, care must be taken to ensure that findings are not merely artefacts due to data selection or inherent bias. To differentiate meaningful patterns from spurious ones, it is important to stress test the findings through different selection criteria or across non-overlapping data sources.

Regression analysis

When investigating correlations among variables, a classic method is regression, which estimates how one set of variables explains variation in an outcome of interest. Regression can be used to test explicit hypotheses or predict outcomes. For example, researchers have investigated whether a paper’s novelty predicts its citation impact 172 . Adding additional control variables to the regression, one can further examine the robustness of the focal relationship.

Although regression analysis is useful for hypothesis testing, it bears substantial limitations. If the question one wishes to ask concerns a ‘causal’ rather than a correlational relationship, regression is poorly suited to the task as it is impossible to control for all the confounding factors. Failing to account for such ‘omitted variables’ can bias the regression coefficient estimates and lead to spurious interpretations. Further, regression models often have low goodness of fit (small R 2 ), indicating that the variables considered explain little of the outcome variation. As regressions typically focus on a specific relationship in simple functional forms, regressions tend to emphasize interpretability rather than overall predictability. The advent of predictive approaches powered by large-scale datasets and novel computational techniques offers new opportunities for modelling complex relationships with stronger predictive power.

Mechanistic models

Mechanistic modelling is an important approach to explaining empirical regularities, drawing from methods primarily used in physics. Such models predict macro-level regularities of a system by modelling micro-level interactions among basic elements with interpretable and modifiable formulars. While theoretical by nature, mechanistic models in the science of science are often empirically grounded, and this approach has developed together with the advent of large-scale, high-resolution data.

Simplicity is the core value of a mechanistic model. Consider for example, why citations follow a fat-tailed distribution. de Solla Price modelled the citing behaviour as a cumulative advantage process on a growing citation network 159 and found that if the probability a paper is cited grows linearly with its existing citations, the resulting distribution would follow a power law, broadly aligned with empirical observations. The model is intentionally simplified, ignoring myriad factors. Yet the simple cumulative advantage process is by itself sufficient in explaining a power law distribution of citations. In this way, mechanistic models can help to reveal key mechanisms that can explain observed patterns.

Moreover, mechanistic models can be refined as empirical evidence evolves. For example, later investigations showed that citation distributions are better characterized as log-normal 156 , 173 , prompting researchers to introduce a fitness parameter to encapsulate the inherent differences in papers’ ability to attract citations 174 , 175 . Further, older papers are less likely to be cited than expected 176 , 177 , 178 , motivating more recent models 20 to introduce an additional aging effect 179 . By combining the cumulative advantage, fitness and aging effects, one can already achieve substantial predictive power not just for the overall properties of the system but also the citation dynamics of individual papers 20 .

In addition to citations, mechanistic models have been developed to understand the formation of collaborations 136 , 180 , 181 , 182 , 183 , knowledge discovery and diffusion 184 , 185 , topic selection 186 , 187 , career dynamics 30 , 31 , 188 , 189 , the growth of scientific fields 190 and the dynamics of failure in science and other domains 178 .

At the same time, some observers have argued that mechanistic models are too simplistic to capture the essence of complex real-world problems 191 . While it has been a cornerstone for the natural sciences, representing social phenomena in a limited set of mathematical equations may miss complexities and heterogeneities that make social phenomena interesting in the first place. Such concerns are not unique to the science of science, as they represent a broader theme in computational social sciences 192 , 193 , ranging from social networks 194 , 195 to human mobility 196 , 197 to epidemics 198 , 199 . Other observers have questioned the practical utility of mechanistic models and whether they can be used to guide decisions and devise actionable policies. Nevertheless, despite these limitations, several complex phenomena in the science of science are well captured by simple mechanistic models, showing a high degree of regularity beneath complex interacting systems and providing powerful insights about the nature of science. Mixing such modelling with other methods could be particularly fruitful in future investigations.

Machine learning

The science of science seeks in part to forecast promising directions for scientific research 7 , 44 . In recent years, machine learning methods have substantially advanced predictive capabilities 200 , 201 and are playing increasingly important parts in the science of science. In contrast to the previous methods, machine learning does not emphasize hypotheses or theories. Rather, it leverages complex relationships in data and optimizes goodness of fit to make predictions and categorizations.

Traditional machine learning models include supervised, semi-supervised and unsupervised learning. The model choice depends on data availability and the research question, ranging from supervised models for citation prediction 202 , 203 to unsupervised models for community detection 204 . Take for example mappings of scientific knowledge 94 , 205 , 206 . The unsupervised method applies network clustering algorithms to map the structures of science. Related visualization tools make sense of clusters from the underlying network, allowing observers to see the organization, interactions and evolution of scientific knowledge. More recently, supervised learning, and deep neural networks in particular, have witnessed especially rapid developments 207 . Neural networks can generate high-dimensional representations of unstructured data such as images and texts, which encode complex properties difficult for human experts to perceive.

Take text analysis as an example. A recent study 52 utilizes 3.3 million paper abstracts in materials science to predict the thermoelectric properties of materials. The intuition is that the words currently used to describe a material may predict its hitherto undiscovered properties (Fig. 2 ). Compared with a random material, the materials predicted by the model are eight times more likely to be reported as thermoelectric in the next 5 years, suggesting that machine learning has the potential to substantially speed up knowledge discovery, especially as data continue to grow in scale and scope. Indeed, predicting the direction of new discoveries represents one of the most promising avenues for machine learning models, with neural networks being applied widely to biology 208 , physics 209 , 210 , mathematics 211 , chemistry 212 , medicine 213 and clinical applications 214 . Neural networks also offer a quantitative framework to probe the characteristics of creative products ranging from scientific papers 53 , journals 215 , organizations 148 , to paintings and movies 32 . Neural networks can also help to predict the reproducibility of papers from a variety of disciplines at scale 53 , 216 .

This figure illustrates the word2vec skip-gram methods 52 , where the goal is to predict useful properties of materials using previous scientific literature. a , The architecture and training process of the word2vec skip-gram model, where the 3-layer, fully connected neural network learns the 200-dimensional representation (hidden layer) from the sparse vector for each word and its context in the literature (input layer). b , The top two principal components of the word embedding. Materials with similar features are close in the 2D space, allowing prediction of a material’s properties. Different targeted words are shown in different colours. Reproduced with permission from ref. 52 , Springer Nature Ltd.

While machine learning can offer high predictive accuracy, successful applications to the science of science face challenges, particularly regarding interpretability. Researchers may value transparent and interpretable findings for how a given feature influences an outcome, rather than a black-box model. The lack of interpretability also raises concerns about bias and fairness. In predicting reproducible patterns from data, machine learning models inevitably include and reproduce biases embedded in these data, often in non-transparent ways. The fairness of machine learning 217 is heavily debated in applications ranging from the criminal justice system to hiring processes. Effective and responsible use of machine learning in the science of science therefore requires thoughtful partnership between humans and machines 53 to build a reliable system accessible to scrutiny and modification.

Causal approaches

The preceding methods can reveal core facts about the workings of science and develop predictive capacity. Yet, they fail to capture causal relationships, which are particularly useful in assessing policy interventions. For example, how can we test whether a science policy boosts or hinders the performance of individuals, teams or institutions? The overarching idea of causal approaches is to construct some counterfactual world where two groups are identical to each other except that one group experiences a treatment that the other group does not.

Towards causation

Before engaging in causal approaches, it is useful to first consider the interpretative challenges of observational data. As observational data emerge from mechanisms that are not fully known or measured, an observed correlation may be driven by underlying forces that were not accounted for in the analysis. This challenge makes causal inference fundamentally difficult in observational data. An awareness of this issue is the first step in confronting it. It further motivates intermediate empirical approaches, including the use of matching strategies and fixed effects, that can help to confront (although not fully eliminate) the inference challenge. We first consider these approaches before turning to more fully causal methods.

Matching. Matching utilizes rich information to construct a control group that is similar to the treatment group on as many observable characteristics as possible before the treatment group is exposed to the treatment. Inferences can then be made by comparing the treatment and the matched control groups. Exact matching applies to categorical values, such as country, gender, discipline or affiliation 35 , 218 . Coarsened exact matching considers percentile bins of continuous variables and matches observations in the same bin 133 . Propensity score matching estimates the probability of receiving the ‘treatment’ on the basis of the controlled variables and uses the estimates to match treatment and control groups, which reduces the matching task from comparing the values of multiple covariates to comparing a single value 24 , 219 . Dynamic matching is useful for longitudinally matching variables that change over time 220 , 221 .

Fixed effects. Fixed effects are a powerful and now standard tool in controlling for confounders. A key requirement for using fixed effects is that there are multiple observations on the same subject or entity (person, field, institution and so on) 222 , 223 , 224 . The fixed effect works as a dummy variable that accounts for the role of any fixed characteristic of that entity. Consider the finding where gender-diverse teams produce higher-impact papers than same-gender teams do 225 . A confounder may be that individuals who tend to write high-impact papers may also be more likely to work in gender-diverse teams. By including individual fixed effects, one accounts for any fixed characteristics of individuals (such as IQ, cultural background or previous education) that might drive the relationship of interest.

In sum, matching and fixed effects methods reduce potential sources of bias in interpreting relationships between variables. Yet, confounders may persist in these studies. For instance, fixed effects do not control for unobserved factors that change with time within the given entity (for example, access to funding or new skills). Identifying casual effects convincingly will then typically require distinct research methods that we turn to next.

Quasi-experiments

Researchers in economics and other fields have developed a range of quasi-experimental methods to construct treatment and control groups. The key idea here is exploiting randomness from external events that differentially expose subjects to a particular treatment. Here we review three quasi-experimental methods: difference-in-differences, instrumental variables and regression discontinuity (Fig. 3 ).

a – c , This figure presents illustrations of ( a ) differences-in-differences, ( b ) instrumental variables and ( c ) regression discontinuity methods. The solid line in b represents causal links and the dashed line represents the relationships that are not allowed, if the IV method is to produce causal inference.

Difference-in-differences. Difference-in-difference regression (DiD) investigates the effect of an unexpected event, comparing the affected group (the treated group) with an unaffected group (the control group). The control group is intended to provide the counterfactual path—what would have happened were it not for the unexpected event. Ideally, the treated and control groups are on virtually identical paths before the treatment event, but DiD can also work if the groups are on parallel paths (Fig. 3a ). For example, one study 226 examines how the premature death of superstar scientists affects the productivity of their previous collaborators. The control group are collaborators of superstars who did not die in the time frame. The two groups do not show significant differences in publications before a death event, yet upon the death of a star scientist, the treated collaborators on average experience a 5–8% decline in their quality-adjusted publication rates compared with the control group. DiD has wide applicability in the science of science, having been used to analyse the causal effects of grant design 24 , access costs to previous research 155 , 227 , university technology transfer policies 154 , intellectual property 228 , citation practices 229 , evolution of fields 221 and the impacts of paper retractions 230 , 231 , 232 . The DiD literature has grown especially rapidly in the field of economics, with substantial recent refinements 233 , 234 .

Instrumental variables. Another quasi-experimental approach utilizes ‘instrumental variables’ (IV). The goal is to determine the causal influence of some feature X on some outcome Y by using a third, instrumental variable. This instrumental variable is a quasi-random event that induces variation in X and, except for its impact through X , has no other effect on the outcome Y (Fig. 3b ). For example, consider a study of astronomy that seeks to understand how telescope time affects career advancement 235 . Here, one cannot simply look at the correlation between telescope time and career outcomes because many confounds (such as talent or grit) may influence both telescope time and career opportunities. Now consider the weather as an instrumental variable. Cloudy weather will, at random, reduce an astronomer’s observational time. Yet, the weather on particular nights is unlikely to correlate with a scientist’s innate qualities. The weather can then provide an instrumental variable to reveal a causal relationship between telescope time and career outcomes. Instrumental variables have been used to study local peer effects in research 151 , the impact of gender composition in scientific committees 236 , patents on future innovation 237 and taxes on inventor mobility 238 .

Regression discontinuity. In regression discontinuity, policies with an arbitrary threshold for receiving some benefit can be used to construct treatment and control groups (Fig. 3c ). Take the funding paylines for grant proposals as an example. Proposals with scores increasingly close to the payline are increasingly similar in their both observable and unobservable characteristics, yet only those projects with scores above the payline receive the funding. For example, a study 110 examines the effect of winning an early-career grant on the probability of winning a later, mid-career grant. The probability has a discontinuous jump across the initial grant’s payline, providing the treatment and control groups needed to estimate the causal effect of receiving a grant. This example utilizes the ‘sharp’ regression discontinuity that assumes treatment status to be fully determined by the cut-off. If we assume treatment status is only partly determined by the cut-off, we can use ‘fuzzy’ regression discontinuity designs. Here the probability of receiving a grant is used to estimate the future outcome 11 , 110 , 239 , 240 , 241 .

Although quasi-experiments are powerful tools, they face their own limitations. First, these approaches identify causal effects within a specific context and often engage small numbers of observations. How representative the samples are for broader populations or contexts is typically left as an open question. Second, the validity of the causal design is typically not ironclad. Researchers usually conduct different robustness checks to verify whether observable confounders have significant differences between the treated and control groups, before treatment. However, unobservable features may still differ between treatment and control groups. The quality of instrumental variables and the specific claim that they have no effect on the outcome except through the variable of interest, is also difficult to assess. Ultimately, researchers must rely partly on judgement to tell whether appropriate conditions are met for causal inference.

This section emphasized popular econometric approaches to causal inference. Other empirical approaches, such as graphical causal modelling 242 , 243 , also represent an important stream of work on assessing causal relationships. Such approaches usually represent causation as a directed acyclic graph, with nodes as variables and arrows between them as suspected causal relationships. In the science of science, the directed acyclic graph approach has been applied to quantify the causal effect of journal impact factor 244 and gender or racial bias 245 on citations. Graphical causal modelling has also triggered discussions on strengths and weaknesses compared to the econometrics methods 246 , 247 .

Experiments

In contrast to quasi-experimental approaches, laboratory and field experiments conduct direct randomization in assigning treatment and control groups. These methods engage explicitly in the data generation process, manipulating interventions to observe counterfactuals. These experiments are crafted to study mechanisms of specific interest and, by designing the experiment and formally randomizing, can produce especially rigorous causal inference.

Laboratory experiments. Laboratory experiments build counterfactual worlds in well-controlled laboratory environments. Researchers randomly assign participants to the treatment or control group and then manipulate the laboratory conditions to observe different outcomes in the two groups. For example, consider laboratory experiments on team performance and gender composition 144 , 248 . The researchers randomly assign participants into groups to perform tasks such as solving puzzles or brainstorming. Teams with a higher proportion of women are found to perform better on average, offering evidence that gender diversity is causally linked to team performance. Laboratory experiments can allow researchers to test forces that are otherwise hard to observe, such as how competition influences creativity 249 . Laboratory experiments have also been used to evaluate how journal impact factors shape scientists’ perceptions of rewards 250 and gender bias in hiring 251 .

Laboratory experiments allow for precise control of settings and procedures to isolate causal effects of interest. However, participants may behave differently in synthetic environments than in real-world settings, raising questions about the generalizability and replicability of the results 252 , 253 , 254 . To assess causal effects in real-world settings, researcher use randomized controlled trials.

Randomized controlled trials. A randomized controlled trial (RCT), or field experiment, is a staple for causal inference across a wide range of disciplines. RCTs randomly assign participants into the treatment and control conditions 255 and can be used not only to assess mechanisms but also to test real-world interventions such as policy change. The science of science has witnessed growing use of RCTs. For instance, a field experiment 146 investigated whether lower search costs for collaborators increased collaboration in grant applications. The authors randomly allocated principal investigators to face-to-face sessions in a medical school, and then measured participants’ chance of writing a grant proposal together. RCTs have also offered rich causal insights on peer review 256 , 257 , 258 , 259 , 260 and gender bias in science 261 , 262 , 263 .

While powerful, RCTs are difficult to conduct in the science of science, mainly for two reasons. The first concerns potential risks in a policy intervention. For instance, while randomizing funding across individuals could generate crucial causal insights for funders, it may also inadvertently harm participants’ careers 264 . Second, key questions in the science of science often require a long-time horizon to trace outcomes, which makes RCTs costly. It also raises the difficulty of replicating findings. A relative advantage of the quasi-experimental methods discussed earlier is that one can identify causal effects over potentially long periods of time in the historical record. On the other hand, quasi-experiments must be found as opposed to designed, and they often are not available for many questions of interest. While the best approaches are context dependent, a growing community of researchers is building platforms to facilitate RCTs for the science of science, aiming to lower their costs and increase their scale. Performing RCTs in partnership with science institutions can also contribute to timely, policy-relevant research that may substantially improve science decision-making and investments.

Research in the science of science has been empowered by the growth of high-scale data, new measurement approaches and an expanding range of empirical methods. These tools provide enormous capacity to test conceptual frameworks about science, discover factors impacting scientific productivity, predict key scientific outcomes and design policies that better facilitate future scientific progress. A careful appreciation of empirical techniques can help researchers to choose effective tools for questions of interest and propel the field. A better and broader understanding of these methodologies may also build bridges across diverse research communities, facilitating communication and collaboration, and better leveraging the value of diverse perspectives. The science of science is about turning scientific methods on the nature of science itself. The fruits of this work, with time, can guide researchers and research institutions to greater progress in discovery and understanding across the landscape of scientific inquiry.

Bush, V . S cience–the Endless Frontier: A Report to the President on a Program for Postwar Scientific Research (National Science Foundation, 1990).

Mokyr, J. The Gifts of Athena (Princeton Univ. Press, 2011).

Jones, B. F. in Rebuilding the Post-Pandemic Economy (eds Kearney, M. S. & Ganz, A.) 272–310 (Aspen Institute Press, 2021).

Wang, D. & Barabási, A.-L. The Science of Science (Cambridge Univ. Press, 2021).

Fortunato, S. et al. Science of science. Science 359 , eaao0185 (2018).

Article   PubMed   PubMed Central   Google Scholar  

Azoulay, P. et al. Toward a more scientific science. Science 361 , 1194–1197 (2018).

Article   PubMed   Google Scholar  

Clauset, A., Larremore, D. B. & Sinatra, R. Data-driven predictions in the science of science. Science 355 , 477–480 (2017).

Article   CAS   PubMed   Google Scholar  

Zeng, A. et al. The science of science: from the perspective of complex systems. Phys. Rep. 714 , 1–73 (2017).

Article   Google Scholar  

Lin, Z., Yin. Y., Liu, L. & Wang, D. SciSciNet: a large-scale open data lake for the science of science research. Sci. Data, https://doi.org/10.1038/s41597-023-02198-9 (2023).

Ahmadpoor, M. & Jones, B. F. The dual frontier: patented inventions and prior scientific advance. Science 357 , 583–587 (2017).

Azoulay, P., Graff Zivin, J. S., Li, D. & Sampat, B. N. Public R&D investments and private-sector patenting: evidence from NIH funding rules. Rev. Econ. Stud. 86 , 117–152 (2019).

Yin, Y., Dong, Y., Wang, K., Wang, D. & Jones, B. F. Public use and public funding of science. Nat. Hum. Behav. 6 , 1344–1350 (2022).

Merton, R. K. The Sociology of Science: Theoretical and Empirical Investigations (Univ. Chicago Press, 1973).

Kuhn, T. The Structure of Scientific Revolutions (Princeton Univ. Press, 2021).

Uzzi, B., Mukherjee, S., Stringer, M. & Jones, B. Atypical combinations and scientific impact. Science 342 , 468–472 (2013).

Zuckerman, H. Scientific Elite: Nobel Laureates in the United States (Transaction Publishers, 1977).

Wu, L., Wang, D. & Evans, J. A. Large teams develop and small teams disrupt science and technology. Nature 566 , 378–382 (2019).

Wuchty, S., Jones, B. F. & Uzzi, B. The increasing dominance of teams in production of knowledge. Science 316 , 1036–1039 (2007).

Foster, J. G., Rzhetsky, A. & Evans, J. A. Tradition and innovation in scientists’ research strategies. Am. Sociol. Rev. 80 , 875–908 (2015).

Wang, D., Song, C. & Barabási, A.-L. Quantifying long-term scientific impact. Science 342 , 127–132 (2013).

Clauset, A., Arbesman, S. & Larremore, D. B. Systematic inequality and hierarchy in faculty hiring networks. Sci. Adv. 1 , e1400005 (2015).

Ma, A., Mondragón, R. J. & Latora, V. Anatomy of funded research in science. Proc. Natl Acad. Sci. USA 112 , 14760–14765 (2015).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ma, Y. & Uzzi, B. Scientific prize network predicts who pushes the boundaries of science. Proc. Natl Acad. Sci. USA 115 , 12608–12615 (2018).

Azoulay, P., Graff Zivin, J. S. & Manso, G. Incentives and creativity: evidence from the academic life sciences. RAND J. Econ. 42 , 527–554 (2011).

Schor, S. & Karten, I. Statistical evaluation of medical journal manuscripts. JAMA 195 , 1123–1128 (1966).

Platt, J. R. Strong inference: certain systematic methods of scientific thinking may produce much more rapid progress than others. Science 146 , 347–353 (1964).

Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2 , e124 (2005).

Simonton, D. K. Career landmarks in science: individual differences and interdisciplinary contrasts. Dev. Psychol. 27 , 119 (1991).

Way, S. F., Morgan, A. C., Clauset, A. & Larremore, D. B. The misleading narrative of the canonical faculty productivity trajectory. Proc. Natl Acad. Sci. USA 114 , E9216–E9223 (2017).

Sinatra, R., Wang, D., Deville, P., Song, C. & Barabási, A.-L. Quantifying the evolution of individual scientific impact. Science 354 , aaf5239 (2016).

Liu, L. et al. Hot streaks in artistic, cultural, and scientific careers. Nature 559 , 396–399 (2018).

Liu, L., Dehmamy, N., Chown, J., Giles, C. L. & Wang, D. Understanding the onset of hot streaks across artistic, cultural, and scientific careers. Nat. Commun. 12 , 5392 (2021).

Squazzoni, F. et al. Peer review and gender bias: a study on 145 scholarly journals. Sci. Adv. 7 , eabd0299 (2021).

Hofstra, B. et al. The diversity–innovation paradox in science. Proc. Natl Acad. Sci. USA 117 , 9284–9291 (2020).

Huang, J., Gates, A. J., Sinatra, R. & Barabási, A.-L. Historical comparison of gender inequality in scientific careers across countries and disciplines. Proc. Natl Acad. Sci. USA 117 , 4609–4616 (2020).

Gläser, J. & Laudel, G. Governing science: how science policy shapes research content. Eur. J. Sociol. 57 , 117–168 (2016).

Stephan, P. E. How Economics Shapes Science (Harvard Univ. Press, 2012).

Garfield, E. & Sher, I. H. New factors in the evaluation of scientific literature through citation indexing. Am. Doc. 14 , 195–201 (1963).

Article   CAS   Google Scholar  

de Solla Price, D. J. Networks of scientific papers. Science 149 , 510–515 (1965).

Etzkowitz, H., Kemelgor, C. & Uzzi, B. Athena Unbound: The Advancement of Women in Science and Technology (Cambridge Univ. Press, 2000).

Simonton, D. K. Scientific Genius: A Psychology of Science (Cambridge Univ. Press, 1988).

Khabsa, M. & Giles, C. L. The number of scholarly documents on the public web. PLoS ONE 9 , e93949 (2014).

Xia, F., Wang, W., Bekele, T. M. & Liu, H. Big scholarly data: a survey. IEEE Trans. Big Data 3 , 18–35 (2017).

Evans, J. A. & Foster, J. G. Metaknowledge. Science 331 , 721–725 (2011).

Milojević, S. Quantifying the cognitive extent of science. J. Informetr. 9 , 962–973 (2015).

Rzhetsky, A., Foster, J. G., Foster, I. T. & Evans, J. A. Choosing experiments to accelerate collective discovery. Proc. Natl Acad. Sci. USA 112 , 14569–14574 (2015).

Poncela-Casasnovas, J., Gerlach, M., Aguirre, N. & Amaral, L. A. Large-scale analysis of micro-level citation patterns reveals nuanced selection criteria. Nat. Hum. Behav. 3 , 568–575 (2019).

Hardwicke, T. E. et al. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. R. Soc. Open Sci. 5 , 180448 (2018).

Nagaraj, A., Shears, E. & de Vaan, M. Improving data access democratizes and diversifies science. Proc. Natl Acad. Sci. USA 117 , 23490–23498 (2020).

Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B. & Squazzoni, F. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nat. Commun. 10 , 322 (2019).

Tran, D. et al. An open review of open review: a critical analysis of the machine learning conference review process. Preprint at https://doi.org/10.48550/arXiv.2010.05137 (2020).

Tshitoyan, V. et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 571 , 95–98 (2019).

Yang, Y., Wu, Y. & Uzzi, B. Estimating the deep replicability of scientific findings using human and artificial intelligence. Proc. Natl Acad. Sci. USA 117 , 10762–10768 (2020).

Mukherjee, S., Uzzi, B., Jones, B. & Stringer, M. A new method for identifying recombinations of existing knowledge associated with high‐impact innovation. J. Prod. Innov. Manage. 33 , 224–236 (2016).

Leahey, E., Beckman, C. M. & Stanko, T. L. Prominent but less productive: the impact of interdisciplinarity on scientists’ research. Adm. Sci. Q. 62 , 105–139 (2017).

Sauermann, H. & Haeussler, C. Authorship and contribution disclosures. Sci. Adv. 3 , e1700404 (2017).

Oliveira, D. F. M., Ma, Y., Woodruff, T. K. & Uzzi, B. Comparison of National Institutes of Health grant amounts to first-time male and female principal investigators. JAMA 321 , 898–900 (2019).

Yang, Y., Chawla, N. V. & Uzzi, B. A network’s gender composition and communication pattern predict women’s leadership success. Proc. Natl Acad. Sci. USA 116 , 2033–2038 (2019).

Way, S. F., Larremore, D. B. & Clauset, A. Gender, productivity, and prestige in computer science faculty hiring networks. In Proc. 25th International Conference on World Wide Web 1169–1179. (ACM 2016)

Malmgren, R. D., Ottino, J. M. & Amaral, L. A. N. The role of mentorship in protege performance. Nature 465 , 622–626 (2010).

Ma, Y., Mukherjee, S. & Uzzi, B. Mentorship and protégé success in STEM fields. Proc. Natl Acad. Sci. USA 117 , 14077–14083 (2020).

Börner, K. et al. Skill discrepancies between research, education, and jobs reveal the critical need to supply soft skills for the data economy. Proc. Natl Acad. Sci. USA 115 , 12630–12637 (2018).

Biasi, B. & Ma, S. The Education-Innovation Gap (National Bureau of Economic Research Working papers, 2020).

Bornmann, L. Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. J. Informetr. 8 , 895–903 (2014).

Cleary, E. G., Beierlein, J. M., Khanuja, N. S., McNamee, L. M. & Ledley, F. D. Contribution of NIH funding to new drug approvals 2010–2016. Proc. Natl Acad. Sci. USA 115 , 2329–2334 (2018).

Spector, J. M., Harrison, R. S. & Fishman, M. C. Fundamental science behind today’s important medicines. Sci. Transl. Med. 10 , eaaq1787 (2018).

Haunschild, R. & Bornmann, L. How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics 110 , 1209–1216 (2017).

Yin, Y., Gao, J., Jones, B. F. & Wang, D. Coevolution of policy and science during the pandemic. Science 371 , 128–130 (2021).

Sugimoto, C. R., Work, S., Larivière, V. & Haustein, S. Scholarly use of social media and altmetrics: a review of the literature. J. Assoc. Inf. Sci. Technol. 68 , 2037–2062 (2017).

Dunham, I. Human genes: time to follow the roads less traveled? PLoS Biol. 16 , e3000034 (2018).

Kustatscher, G. et al. Understudied proteins: opportunities and challenges for functional proteomics. Nat. Methods 19 , 774–779 (2022).

Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86 , 638 (1979).

Franco, A., Malhotra, N. & Simonovits, G. Publication bias in the social sciences: unlocking the file drawer. Science 345 , 1502–1505 (2014).

Vera-Baceta, M.-A., Thelwall, M. & Kousha, K. Web of Science and Scopus language coverage. Scientometrics 121 , 1803–1813 (2019).

Waltman, L. A review of the literature on citation impact indicators. J. Informetr. 10 , 365–391 (2016).

Garfield, E. & Merton, R. K. Citation Indexing: Its Theory and Application in Science, Technology, and Humanities (Wiley, 1979).

Kelly, B., Papanikolaou, D., Seru, A. & Taddy, M. Measuring Technological Innovation Over the Long Run Report No. 0898-2937 (National Bureau of Economic Research, 2018).

Kogan, L., Papanikolaou, D., Seru, A. & Stoffman, N. Technological innovation, resource allocation, and growth. Q. J. Econ. 132 , 665–712 (2017).

Hall, B. H., Jaffe, A. & Trajtenberg, M. Market value and patent citations. RAND J. Econ. 36 , 16–38 (2005).

Google Scholar  

Yan, E. & Ding, Y. Applying centrality measures to impact analysis: a coauthorship network analysis. J. Am. Soc. Inf. Sci. Technol. 60 , 2107–2118 (2009).

Radicchi, F., Fortunato, S., Markines, B. & Vespignani, A. Diffusion of scientific credits and the ranking of scientists. Phys. Rev. E 80 , 056103 (2009).

Bollen, J., Rodriquez, M. A. & Van de Sompel, H. Journal status. Scientometrics 69 , 669–687 (2006).

Bergstrom, C. T., West, J. D. & Wiseman, M. A. The eigenfactor™ metrics. J. Neurosci. 28 , 11433–11434 (2008).

Cronin, B. & Sugimoto, C. R. Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact (MIT Press, 2014).

Hicks, D., Wouters, P., Waltman, L., De Rijcke, S. & Rafols, I. Bibliometrics: the Leiden Manifesto for research metrics. Nature 520 , 429–431 (2015).

Catalini, C., Lacetera, N. & Oettl, A. The incidence and role of negative citations in science. Proc. Natl Acad. Sci. USA 112 , 13823–13826 (2015).

Alcacer, J. & Gittelman, M. Patent citations as a measure of knowledge flows: the influence of examiner citations. Rev. Econ. Stat. 88 , 774–779 (2006).

Ding, Y. et al. Content‐based citation analysis: the next generation of citation analysis. J. Assoc. Inf. Sci. Technol. 65 , 1820–1833 (2014).

Teufel, S., Siddharthan, A. & Tidhar, D. Automatic classification of citation function. In Proc. 2006 Conference on Empirical Methods in Natural Language Processing, 103–110 (Association for Computational Linguistics 2006)

Seeber, M., Cattaneo, M., Meoli, M. & Malighetti, P. Self-citations as strategic response to the use of metrics for career decisions. Res. Policy 48 , 478–491 (2019).

Pendlebury, D. A. The use and misuse of journal metrics and other citation indicators. Arch. Immunol. Ther. Exp. 57 , 1–11 (2009).

Biagioli, M. Watch out for cheats in citation game. Nature 535 , 201 (2016).

Jo, W. S., Liu, L. & Wang, D. See further upon the giants: quantifying intellectual lineage in science. Quant. Sci. Stud. 3 , 319–330 (2022).

Boyack, K. W., Klavans, R. & Börner, K. Mapping the backbone of science. Scientometrics 64 , 351–374 (2005).

Gates, A. J., Ke, Q., Varol, O. & Barabási, A.-L. Nature’s reach: narrow work has broad impact. Nature 575 , 32–34 (2019).

Börner, K., Penumarthy, S., Meiss, M. & Ke, W. Mapping the diffusion of scholarly knowledge among major US research institutions. Scientometrics 68 , 415–426 (2006).

King, D. A. The scientific impact of nations. Nature 430 , 311–316 (2004).

Pan, R. K., Kaski, K. & Fortunato, S. World citation and collaboration networks: uncovering the role of geography in science. Sci. Rep. 2 , 902 (2012).

Jaffe, A. B., Trajtenberg, M. & Henderson, R. Geographic localization of knowledge spillovers as evidenced by patent citations. Q. J. Econ. 108 , 577–598 (1993).

Funk, R. J. & Owen-Smith, J. A dynamic network measure of technological change. Manage. Sci. 63 , 791–817 (2017).

Yegros-Yegros, A., Rafols, I. & D’este, P. Does interdisciplinary research lead to higher citation impact? The different effect of proximal and distal interdisciplinarity. PLoS ONE 10 , e0135095 (2015).

Larivière, V., Haustein, S. & Börner, K. Long-distance interdisciplinarity leads to higher scientific impact. PLoS ONE 10 , e0122565 (2015).

Fleming, L., Greene, H., Li, G., Marx, M. & Yao, D. Government-funded research increasingly fuels innovation. Science 364 , 1139–1141 (2019).

Bowen, A. & Casadevall, A. Increasing disparities between resource inputs and outcomes, as measured by certain health deliverables, in biomedical research. Proc. Natl Acad. Sci. USA 112 , 11335–11340 (2015).

Li, D., Azoulay, P. & Sampat, B. N. The applied value of public investments in biomedical research. Science 356 , 78–81 (2017).

Lehman, H. C. Age and Achievement (Princeton Univ. Press, 2017).

Simonton, D. K. Creative productivity: a predictive and explanatory model of career trajectories and landmarks. Psychol. Rev. 104 , 66 (1997).

Duch, J. et al. The possible role of resource requirements and academic career-choice risk on gender differences in publication rate and impact. PLoS ONE 7 , e51332 (2012).

Wang, Y., Jones, B. F. & Wang, D. Early-career setback and future career impact. Nat. Commun. 10 , 4331 (2019).

Bol, T., de Vaan, M. & van de Rijt, A. The Matthew effect in science funding. Proc. Natl Acad. Sci. USA 115 , 4887–4890 (2018).

Jones, B. F. Age and great invention. Rev. Econ. Stat. 92 , 1–14 (2010).

Newman, M. Networks (Oxford Univ. Press, 2018).

Mazloumian, A., Eom, Y.-H., Helbing, D., Lozano, S. & Fortunato, S. How citation boosts promote scientific paradigm shifts and nobel prizes. PLoS ONE 6 , e18975 (2011).

Hirsch, J. E. An index to quantify an individual’s scientific research output. Proc. Natl Acad. Sci. USA 102 , 16569–16572 (2005).

Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E. & Herrera, F. h-index: a review focused in its variants, computation and standardization for different scientific fields. J. Informetr. 3 , 273–289 (2009).

Egghe, L. An improvement of the h-index: the g-index. ISSI Newsl. 2 , 8–9 (2006).

Kaur, J., Radicchi, F. & Menczer, F. Universality of scholarly impact metrics. J. Informetr. 7 , 924–932 (2013).

Majeti, D. et al. Scholar plot: design and evaluation of an information interface for faculty research performance. Front. Res. Metr. Anal. 4 , 6 (2020).

Sidiropoulos, A., Katsaros, D. & Manolopoulos, Y. Generalized Hirsch h-index for disclosing latent facts in citation networks. Scientometrics 72 , 253–280 (2007).

Jones, B. F. & Weinberg, B. A. Age dynamics in scientific creativity. Proc. Natl Acad. Sci. USA 108 , 18910–18914 (2011).

Dennis, W. Age and productivity among scientists. Science 123 , 724–725 (1956).

Sanyal, D. K., Bhowmick, P. K. & Das, P. P. A review of author name disambiguation techniques for the PubMed bibliographic database. J. Inf. Sci. 47 , 227–254 (2021).

Haak, L. L., Fenner, M., Paglione, L., Pentz, E. & Ratner, H. ORCID: a system to uniquely identify researchers. Learn. Publ. 25 , 259–264 (2012).

Malmgren, R. D., Ottino, J. M. & Amaral, L. A. N. The role of mentorship in protégé performance. Nature 465 , 662–667 (2010).

Oettl, A. Reconceptualizing stars: scientist helpfulness and peer performance. Manage. Sci. 58 , 1122–1140 (2012).

Morgan, A. C. et al. The unequal impact of parenthood in academia. Sci. Adv. 7 , eabd1996 (2021).

Morgan, A. C. et al. Socioeconomic roots of academic faculty. Nat. Hum. Behav. 6 , 1625–1633 (2022).

San Francisco Declaration on Research Assessment (DORA) (American Society for Cell Biology, 2012).

Falk‐Krzesinski, H. J. et al. Advancing the science of team science. Clin. Transl. Sci. 3 , 263–266 (2010).

Cooke, N. J. et al. Enhancing the Effectiveness of Team Science (National Academies Press, 2015).

Börner, K. et al. A multi-level systems perspective for the science of team science. Sci. Transl. Med. 2 , 49cm24 (2010).

Leahey, E. From sole investigator to team scientist: trends in the practice and study of research collaboration. Annu. Rev. Sociol. 42 , 81–100 (2016).

AlShebli, B. K., Rahwan, T. & Woon, W. L. The preeminence of ethnic diversity in scientific collaboration. Nat. Commun. 9 , 5163 (2018).

Hsiehchen, D., Espinoza, M. & Hsieh, A. Multinational teams and diseconomies of scale in collaborative research. Sci. Adv. 1 , e1500211 (2015).

Koning, R., Samila, S. & Ferguson, J.-P. Who do we invent for? Patents by women focus more on women’s health, but few women get to invent. Science 372 , 1345–1348 (2021).

Barabâsi, A.-L. et al. Evolution of the social network of scientific collaborations. Physica A 311 , 590–614 (2002).

Newman, M. E. Scientific collaboration networks. I. Network construction and fundamental results. Phys. Rev. E 64 , 016131 (2001).

Newman, M. E. Scientific collaboration networks. II. Shortest paths, weighted networks, and centrality. Phys. Rev. E 64 , 016132 (2001).

Palla, G., Barabási, A.-L. & Vicsek, T. Quantifying social group evolution. Nature 446 , 664–667 (2007).

Ross, M. B. et al. Women are credited less in science than men. Nature 608 , 135–145 (2022).

Shen, H.-W. & Barabási, A.-L. Collective credit allocation in science. Proc. Natl Acad. Sci. USA 111 , 12325–12330 (2014).

Merton, R. K. Matthew effect in science. Science 159 , 56–63 (1968).

Ni, C., Smith, E., Yuan, H., Larivière, V. & Sugimoto, C. R. The gendered nature of authorship. Sci. Adv. 7 , eabe4639 (2021).

Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N. & Malone, T. W. Evidence for a collective intelligence factor in the performance of human groups. Science 330 , 686–688 (2010).

Feldon, D. F. et al. Postdocs’ lab engagement predicts trajectories of PhD students’ skill development. Proc. Natl Acad. Sci. USA 116 , 20910–20916 (2019).

Boudreau, K. J. et al. A field experiment on search costs and the formation of scientific collaborations. Rev. Econ. Stat. 99 , 565–576 (2017).

Holcombe, A. O. Contributorship, not authorship: use CRediT to indicate who did what. Publications 7 , 48 (2019).

Murray, D. et al. Unsupervised embedding of trajectories captures the latent structure of mobility. Preprint at https://doi.org/10.48550/arXiv.2012.02785 (2020).

Deville, P. et al. Career on the move: geography, stratification, and scientific impact. Sci. Rep. 4 , 4770 (2014).

Edmunds, L. D. et al. Why do women choose or reject careers in academic medicine? A narrative review of empirical evidence. Lancet 388 , 2948–2958 (2016).

Waldinger, F. Peer effects in science: evidence from the dismissal of scientists in Nazi Germany. Rev. Econ. Stud. 79 , 838–861 (2012).

Agrawal, A., McHale, J. & Oettl, A. How stars matter: recruiting and peer effects in evolutionary biology. Res. Policy 46 , 853–867 (2017).

Fiore, S. M. Interdisciplinarity as teamwork: how the science of teams can inform team science. Small Group Res. 39 , 251–277 (2008).

Hvide, H. K. & Jones, B. F. University innovation and the professor’s privilege. Am. Econ. Rev. 108 , 1860–1898 (2018).

Murray, F., Aghion, P., Dewatripont, M., Kolev, J. & Stern, S. Of mice and academics: examining the effect of openness on innovation. Am. Econ. J. Econ. Policy 8 , 212–252 (2016).

Radicchi, F., Fortunato, S. & Castellano, C. Universality of citation distributions: toward an objective measure of scientific impact. Proc. Natl Acad. Sci. USA 105 , 17268–17272 (2008).

Waltman, L., van Eck, N. J. & van Raan, A. F. Universality of citation distributions revisited. J. Am. Soc. Inf. Sci. Technol. 63 , 72–77 (2012).

Barabási, A.-L. & Albert, R. Emergence of scaling in random networks. Science 286 , 509–512 (1999).

de Solla Price, D. A general theory of bibliometric and other cumulative advantage processes. J. Am. Soc. Inf. Sci. 27 , 292–306 (1976).

Cole, S. Age and scientific performance. Am. J. Sociol. 84 , 958–977 (1979).

Ke, Q., Ferrara, E., Radicchi, F. & Flammini, A. Defining and identifying sleeping beauties in science. Proc. Natl Acad. Sci. USA 112 , 7426–7431 (2015).

Bornmann, L., de Moya Anegón, F. & Leydesdorff, L. Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE 5 , e13327 (2010).

Mukherjee, S., Romero, D. M., Jones, B. & Uzzi, B. The nearly universal link between the age of past knowledge and tomorrow’s breakthroughs in science and technology: the hotspot. Sci. Adv. 3 , e1601315 (2017).

Packalen, M. & Bhattacharya, J. NIH funding and the pursuit of edge science. Proc. Natl Acad. Sci. USA 117 , 12011–12016 (2020).

Zeng, A., Fan, Y., Di, Z., Wang, Y. & Havlin, S. Fresh teams are associated with original and multidisciplinary research. Nat. Hum. Behav. 5 , 1314–1322 (2021).

Newman, M. E. The structure of scientific collaboration networks. Proc. Natl Acad. Sci. USA 98 , 404–409 (2001).

Larivière, V., Ni, C., Gingras, Y., Cronin, B. & Sugimoto, C. R. Bibliometrics: global gender disparities in science. Nature 504 , 211–213 (2013).

West, J. D., Jacquet, J., King, M. M., Correll, S. J. & Bergstrom, C. T. The role of gender in scholarly authorship. PLoS ONE 8 , e66212 (2013).

Gao, J., Yin, Y., Myers, K. R., Lakhani, K. R. & Wang, D. Potentially long-lasting effects of the pandemic on scientists. Nat. Commun. 12 , 6188 (2021).

Jones, B. F., Wuchty, S. & Uzzi, B. Multi-university research teams: shifting impact, geography, and stratification in science. Science 322 , 1259–1262 (2008).

Chu, J. S. & Evans, J. A. Slowed canonical progress in large fields of science. Proc. Natl Acad. Sci. USA 118 , e2021636118 (2021).

Wang, J., Veugelers, R. & Stephan, P. Bias against novelty in science: a cautionary tale for users of bibliometric indicators. Res. Policy 46 , 1416–1436 (2017).

Stringer, M. J., Sales-Pardo, M. & Amaral, L. A. Statistical validation of a global model for the distribution of the ultimate number of citations accrued by papers published in a scientific journal. J. Assoc. Inf. Sci. Technol. 61 , 1377–1385 (2010).

Bianconi, G. & Barabási, A.-L. Bose-Einstein condensation in complex networks. Phys. Rev. Lett. 86 , 5632 (2001).

Bianconi, G. & Barabási, A.-L. Competition and multiscaling in evolving networks. Europhys. Lett. 54 , 436 (2001).

Yin, Y. & Wang, D. The time dimension of science: connecting the past to the future. J. Informetr. 11 , 608–621 (2017).

Pan, R. K., Petersen, A. M., Pammolli, F. & Fortunato, S. The memory of science: Inflation, myopia, and the knowledge network. J. Informetr. 12 , 656–678 (2018).

Yin, Y., Wang, Y., Evans, J. A. & Wang, D. Quantifying the dynamics of failure across science, startups and security. Nature 575 , 190–194 (2019).

Candia, C. & Uzzi, B. Quantifying the selective forgetting and integration of ideas in science and technology. Am. Psychol. 76 , 1067 (2021).

Milojević, S. Principles of scientific research team formation and evolution. Proc. Natl Acad. Sci. USA 111 , 3984–3989 (2014).

Guimera, R., Uzzi, B., Spiro, J. & Amaral, L. A. N. Team assembly mechanisms determine collaboration network structure and team performance. Science 308 , 697–702 (2005).

Newman, M. E. Coauthorship networks and patterns of scientific collaboration. Proc. Natl Acad. Sci. USA 101 , 5200–5205 (2004).

Newman, M. E. Clustering and preferential attachment in growing networks. Phys. Rev. E 64 , 025102 (2001).

Iacopini, I., Milojević, S. & Latora, V. Network dynamics of innovation processes. Phys. Rev. Lett. 120 , 048301 (2018).

Kuhn, T., Perc, M. & Helbing, D. Inheritance patterns in citation networks reveal scientific memes. Phys. Rev. 4 , 041036 (2014).

Jia, T., Wang, D. & Szymanski, B. K. Quantifying patterns of research-interest evolution. Nat. Hum. Behav. 1 , 0078 (2017).

Zeng, A. et al. Increasing trend of scientists to switch between topics. Nat. Commun. https://doi.org/10.1038/s41467-019-11401-8 (2019).

Siudem, G., Żogała-Siudem, B., Cena, A. & Gagolewski, M. Three dimensions of scientific impact. Proc. Natl Acad. Sci. USA 117 , 13896–13900 (2020).

Petersen, A. M. et al. Reputation and impact in academic careers. Proc. Natl Acad. Sci. USA 111 , 15316–15321 (2014).

Jin, C., Song, C., Bjelland, J., Canright, G. & Wang, D. Emergence of scaling in complex substitutive systems. Nat. Hum. Behav. 3 , 837–846 (2019).

Hofman, J. M. et al. Integrating explanation and prediction in computational social science. Nature 595 , 181–188 (2021).

Lazer, D. et al. Computational social science. Science 323 , 721–723 (2009).

Lazer, D. M. et al. Computational social science: obstacles and opportunities. Science 369 , 1060–1062 (2020).

Albert, R. & Barabási, A.-L. Statistical mechanics of complex networks. Rev. Mod. Phys. 74 , 47 (2002).

Newman, M. E. The structure and function of complex networks. SIAM Rev. 45 , 167–256 (2003).

Song, C., Qu, Z., Blumm, N. & Barabási, A.-L. Limits of predictability in human mobility. Science 327 , 1018–1021 (2010).

Alessandretti, L., Aslak, U. & Lehmann, S. The scales of human mobility. Nature 587 , 402–407 (2020).

Pastor-Satorras, R. & Vespignani, A. Epidemic spreading in scale-free networks. Phys. Rev. Lett. 86 , 3200 (2001).

Pastor-Satorras, R., Castellano, C., Van Mieghem, P. & Vespignani, A. Epidemic processes in complex networks. Rev. Mod. Phys. 87 , 925 (2015).

Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016).

Bishop, C. M. Pattern Recognition and Machine Learning (Springer, 2006).

Dong, Y., Johnson, R. A. & Chawla, N. V. Will this paper increase your h-index? Scientific impact prediction. In Proc. 8th ACM International Conference on Web Search and Data Mining, 149–158 (ACM 2015)

Xiao, S. et al. On modeling and predicting individual paper citation count over time. In IJCAI, 2676–2682 (IJCAI, 2016)

Fortunato, S. Community detection in graphs. Phys. Rep. 486 , 75–174 (2010).

Chen, C. Science mapping: a systematic review of the literature. J. Data Inf. Sci. 2 , 1–40 (2017).

CAS   Google Scholar  

Van Eck, N. J. & Waltman, L. Citation-based clustering of publications using CitNetExplorer and VOSviewer. Scientometrics 111 , 1053–1070 (2017).

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521 , 436–444 (2015).

Senior, A. W. et al. Improved protein structure prediction using potentials from deep learning. Nature 577 , 706–710 (2020).

Krenn, M. & Zeilinger, A. Predicting research trends with semantic and neural networks with an application in quantum physics. Proc. Natl Acad. Sci. USA 117 , 1910–1916 (2020).

Iten, R., Metger, T., Wilming, H., Del Rio, L. & Renner, R. Discovering physical concepts with neural networks. Phys. Rev. Lett. 124 , 010508 (2020).

Guimerà, R. et al. A Bayesian machine scientist to aid in the solution of challenging scientific problems. Sci. Adv. 6 , eaav6971 (2020).

Segler, M. H., Preuss, M. & Waller, M. P. Planning chemical syntheses with deep neural networks and symbolic AI. Nature 555 , 604–610 (2018).

Ryu, J. Y., Kim, H. U. & Lee, S. Y. Deep learning improves prediction of drug–drug and drug–food interactions. Proc. Natl Acad. Sci. USA 115 , E4304–E4311 (2018).

Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172 , 1122–1131.e9 (2018).

Peng, H., Ke, Q., Budak, C., Romero, D. M. & Ahn, Y.-Y. Neural embeddings of scholarly periodicals reveal complex disciplinary organizations. Sci. Adv. 7 , eabb9004 (2021).

Youyou, W., Yang, Y. & Uzzi, B. A discipline-wide investigation of the replicability of psychology papers over the past two decades. Proc. Natl Acad. Sci. USA 120 , e2208863120 (2023).

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. & Galstyan, A. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54 , 1–35 (2021).

Way, S. F., Morgan, A. C., Larremore, D. B. & Clauset, A. Productivity, prominence, and the effects of academic environment. Proc. Natl Acad. Sci. USA 116 , 10729–10733 (2019).

Li, W., Aste, T., Caccioli, F. & Livan, G. Early coauthorship with top scientists predicts success in academic careers. Nat. Commun. 10 , 5170 (2019).

Hendry, D. F., Pagan, A. R. & Sargan, J. D. Dynamic specification. Handb. Econ. 2 , 1023–1100 (1984).

Jin, C., Ma, Y. & Uzzi, B. Scientific prizes and the extraordinary growth of scientific topics. Nat. Commun. 12 , 5619 (2021).

Azoulay, P., Ganguli, I. & Zivin, J. G. The mobility of elite life scientists: professional and personal determinants. Res. Policy 46 , 573–590 (2017).

Slavova, K., Fosfuri, A. & De Castro, J. O. Learning by hiring: the effects of scientists’ inbound mobility on research performance in academia. Organ. Sci. 27 , 72–89 (2016).

Sarsons, H. Recognition for group work: gender differences in academia. Am. Econ. Rev. 107 , 141–145 (2017).

Campbell, L. G., Mehtani, S., Dozier, M. E. & Rinehart, J. Gender-heterogeneous working groups produce higher quality science. PLoS ONE 8 , e79147 (2013).

Azoulay, P., Graff Zivin, J. S. & Wang, J. Superstar extinction. Q. J. Econ. 125 , 549–589 (2010).

Furman, J. L. & Stern, S. Climbing atop the shoulders of giants: the impact of institutions on cumulative research. Am. Econ. Rev. 101 , 1933–1963 (2011).

Williams, H. L. Intellectual property rights and innovation: evidence from the human genome. J. Polit. Econ. 121 , 1–27 (2013).

Rubin, A. & Rubin, E. Systematic Bias in the Progress of Research. J. Polit. Econ. 129 , 2666–2719 (2021).

Lu, S. F., Jin, G. Z., Uzzi, B. & Jones, B. The retraction penalty: evidence from the Web of Science. Sci. Rep. 3 , 3146 (2013).

Jin, G. Z., Jones, B., Lu, S. F. & Uzzi, B. The reverse Matthew effect: consequences of retraction in scientific teams. Rev. Econ. Stat. 101 , 492–506 (2019).

Azoulay, P., Bonatti, A. & Krieger, J. L. The career effects of scandal: evidence from scientific retractions. Res. Policy 46 , 1552–1569 (2017).

Goodman-Bacon, A. Difference-in-differences with variation in treatment timing. J. Econ. 225 , 254–277 (2021).

Callaway, B. & Sant’Anna, P. H. Difference-in-differences with multiple time periods. J. Econ. 225 , 200–230 (2021).

Hill, R. Searching for Superstars: Research Risk and Talent Discovery in Astronomy Working Paper (Massachusetts Institute of Technology, 2019).

Bagues, M., Sylos-Labini, M. & Zinovyeva, N. Does the gender composition of scientific committees matter? Am. Econ. Rev. 107 , 1207–1238 (2017).

Sampat, B. & Williams, H. L. How do patents affect follow-on innovation? Evidence from the human genome. Am. Econ. Rev. 109 , 203–236 (2019).

Moretti, E. & Wilson, D. J. The effect of state taxes on the geographical location of top earners: evidence from star scientists. Am. Econ. Rev. 107 , 1858–1903 (2017).

Jacob, B. A. & Lefgren, L. The impact of research grant funding on scientific productivity. J. Public Econ. 95 , 1168–1177 (2011).

Li, D. Expertise versus bias in evaluation: evidence from the NIH. Am. Econ. J. Appl. Econ. 9 , 60–92 (2017).

Pearl, J. Causal diagrams for empirical research. Biometrika 82 , 669–688 (1995).

Pearl, J. & Mackenzie, D. The Book of Why: The New Science of Cause and Effect (Basic Books, 2018).

Traag, V. A. Inferring the causal effect of journals on citations. Quant. Sci. Stud. 2 , 496–504 (2021).

Traag, V. & Waltman, L. Causal foundations of bias, disparity and fairness. Preprint at https://doi.org/10.48550/arXiv.2207.13665 (2022).

Imbens, G. W. Potential outcome and directed acyclic graph approaches to causality: relevance for empirical practice in economics. J. Econ. Lit. 58 , 1129–1179 (2020).

Heckman, J. J. & Pinto, R. Causality and Econometrics (National Bureau of Economic Research, 2022).

Aggarwal, I., Woolley, A. W., Chabris, C. F. & Malone, T. W. The impact of cognitive style diversity on implicit learning in teams. Front. Psychol. 10 , 112 (2019).

Balietti, S., Goldstone, R. L. & Helbing, D. Peer review and competition in the Art Exhibition Game. Proc. Natl Acad. Sci. USA 113 , 8414–8419 (2016).

Paulus, F. M., Rademacher, L., Schäfer, T. A. J., Müller-Pinzler, L. & Krach, S. Journal impact factor shapes scientists’ reward signal in the prospect of publication. PLoS ONE 10 , e0142537 (2015).

Williams, W. M. & Ceci, S. J. National hiring experiments reveal 2:1 faculty preference for women on STEM tenure track. Proc. Natl Acad. Sci. USA 112 , 5360–5365 (2015).

Collaboration, O. S. Estimating the reproducibility of psychological science. Science 349 , aac4716 (2015).

Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351 , 1433–1436 (2016).

Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2 , 637–644 (2018).

Duflo, E. & Banerjee, A. Handbook of Field Experiments (Elsevier, 2017).

Tomkins, A., Zhang, M. & Heavlin, W. D. Reviewer bias in single versus double-blind peer review. Proc. Natl Acad. Sci. USA 114 , 12708–12713 (2017).

Blank, R. M. The effects of double-blind versus single-blind reviewing: experimental evidence from the American Economic Review. Am. Econ. Rev. 81 , 1041–1067 (1991).

Boudreau, K. J., Guinan, E. C., Lakhani, K. R. & Riedl, C. Looking across and looking beyond the knowledge frontier: intellectual distance, novelty, and resource allocation in science. Manage. Sci. 62 , 2765–2783 (2016).

Lane, J. et al. When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects Working Paper #21-007 (Harvard Business School, 2020).

Teplitskiy, M. et al. Do Experts Listen to Other Experts? Field Experimental Evidence from Scientific Peer Review (Harvard Business School, 2019).

Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L., Graham, M. J. & Handelsman, J. Science faculty’s subtle gender biases favor male students. Proc. Natl Acad. Sci. USA 109 , 16474–16479 (2012).

Forscher, P. S., Cox, W. T., Brauer, M. & Devine, P. G. Little race or gender bias in an experiment of initial review of NIH R01 grant proposals. Nat. Hum. Behav. 3 , 257–264 (2019).

Dennehy, T. C. & Dasgupta, N. Female peer mentors early in college increase women’s positive academic experiences and retention in engineering. Proc. Natl Acad. Sci. USA 114 , 5964–5969 (2017).

Azoulay, P. Turn the scientific method on ourselves. Nature 484 , 31–32 (2012).

Download references

Acknowledgements

The authors thank all members of the Center for Science of Science and Innovation (CSSI) for invaluable comments. This work was supported by the Air Force Office of Scientific Research under award number FA9550-19-1-0354, National Science Foundation grant SBE 1829344, and the Alfred P. Sloan Foundation G-2019-12485.

Author information

Authors and affiliations.

Center for Science of Science and Innovation, Northwestern University, Evanston, IL, USA

Lu Liu, Benjamin F. Jones, Brian Uzzi & Dashun Wang

Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, USA

Kellogg School of Management, Northwestern University, Evanston, IL, USA

College of Information Sciences and Technology, Pennsylvania State University, University Park, PA, USA

National Bureau of Economic Research, Cambridge, MA, USA

Benjamin F. Jones

Brookings Institution, Washington, DC, USA

McCormick School of Engineering, Northwestern University, Evanston, IL, USA

• Dashun Wang

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Dashun Wang .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Human Behaviour thanks Ludo Waltman, Erin Leahey and Sarah Bratt for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Liu, L., Jones, B.F., Uzzi, B. et al. Data, measurement and empirical methods in the science of science. Nat Hum Behav 7 , 1046–1058 (2023). https://doi.org/10.1038/s41562-023-01562-4

Download citation

Received : 30 June 2022

Accepted : 17 February 2023

Published : 01 June 2023

Issue Date : July 2023

DOI : https://doi.org/10.1038/s41562-023-01562-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Rescaling the disruption index reveals the universality of disruption distributions in science.

• Alex J. Yang
• Hongcun Gong
• Sanhong Deng

Scientometrics (2024)

Scientific Data (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Technical Support
• Find My Rep

You are here

Journal of Empirical Research on Human Research Ethics

Preview this book.

• Description
• Aims and Scope
• Editorial Board
• Abstracting / Indexing
• Submission Guidelines

The Journal of Empirical Research on Human Research Ethics (JERHRE) publishes empirical research and reviews of empirical literature on human research ethics. Empirical knowledge translates ethical principles into procedures appropriate to specific cultures, contexts, and research topics.

JERHRE is the only journal in the field of human research ethics dedicated exclusively to empirical research. Its distinguished editorial and advisory board brings a range of expertise and international perspective to provide high-quality double-blind peer-reviewed original articles. Topics of recent articles include the following:

• Communication Issues: Recruitment; Informed consent; Deception; Relationships as a source of data; Community consultation and outreach; Language and meaning across cultures and contexts
• Acquisition and Use of Data: Privacy; Confidentiality; Uses of data
• Risk and Benefit: Risk, wrong and harm; Benefit, incentive, promise of social value; Risk/benefit assessment
• Theory, Method and Design: Validity; Modeling; Equitable treatment of participants; Technology, efficiency and sampling; Beliefs about knowledge
• Other Influences on Research : Research ethics committees; Other institutional-governance influences; Perceptions that influence research; Taboo, questionable and controversial topics of research; Scientific integrity and responsibility; Ethics and politics; Government and agency regulations and policies; Human-research literacy; Education in the responsible conduct of research (RCR)

Institutions and their researchers share concern about the responsible conduct of research (RCR), but can experience difficulty finding common ground around the interpretation of ethical principles and regulations. JERHRE seeks to create collaboration among these stakeholders by stimulating research and disseminating knowledge to foster the intelligent application of ethical principles in research contexts worldwide.

The basic aim of JERHRE is to improve ethical problem solving in human research. Stakeholders in human research grapple with conflict among various standards. Without evidence-based problem solving, many conflicts are unsatisfactorily settled by applying one-size-fits-all interpretation of principles or regulations, or resorting to anecdote as evidence for one or another interpretation. JERHRE creates collaboration among stakeholders, stimulates research, and disseminates knowledge to foster intelligent application of ethical principles in research contexts worldwide.

• BIOETHICSLINE©
• Bibliography of BioEthics
• Cabel’s Directory of Publishing Opportunities in Psychiatry and Psychology
• Clarivate Analytics: Social Science Citation Index
• ETHX on the Web
• Internationale Bibliographie der Geistes- und Sozialwissenschatlichen
• Internationale Bibliographie der Rezensionen Wissenshcaftlicher Literatur
• PubMed: MEDLINE
• SwetsWise Index
• Thomson-Reuters Journal Citation Reports
• Thomson-Reuters Science Citation Index
• Thomson-Reuters Web of Science

The following is intended to guide and expand, not limit, the scope of articles. As the environment of human research evolves, this section will evolve. If your intended contribution does not fit the categories described here, you are invited to consult with the Editor-in-Chief Doug Wassenaar at [email protected]

Manuscript Submission

Manuscripts should be transmitted in Word Document or LaTeX form through the journal’s Sage Track website at http://mc.manuscriptcentral.com/jerhre .

Manuscripts prepared for submission to JERHRE should follow the 6 th edition of the Publication Manual of the American Psychological Association and be written in English.

Manuscripts may be major articles (see Manuscript Types Considered For Submission ), brief reports of preliminary studies, or book reviews, which are usually by invitation of the Book Review Editor. Articles should be concise, useful and relevant to JERHRE ’s target audiences: researchers, members of research ethics committees/IRBs, empirical research ethics scholars/academics, research stakeholders, and research administrators. Articles should be about 4,000 words inclusive, but some flexibility is accommodated if justified. Use of supplemental digital content (SDC) is encouraged if additional detail would be of interest to some readers. Authors may send SDC to be considered for online posting. SDC may include text documents, data sets, replication code, zip files, additional images, and appendices as well as multimedia materials such as audio or video recordings; multimedia requirements are available upon request. For topics typically covered in articles, but not exclusively, fields and disciplines to address, and acceptable approaches to methods and data please read the detailed guide, Manuscript Types Considered for Submission . Authors may contact the editorial office ( [email protected] ) with potential submission ideas if they are in doubt as to whether or not they fit the scope or interests of the journal.

Because manuscripts are anonymize reviewed, the parts of the manuscript that reveal the identity of the author must be separate documents. Any figures and tables should also be transmitted as separate documents. Therefore, in the final preparation for submission, create the following as separate documents:

• Manuscript, minus figures, tables, and any identifying information
• Any figures, tables or supplementary online documents
• Acknowledgements, Author Note, Biographical Sketch(es)
• Cover letter

Please see the document Manuscript Preparation for more information on how to prepare these parts of your article for submission to JERHRE and comply with the journal’s submission requirements.

General Policies

Policies governing JERHRE 's authors, editors, and reviewers are those of the Council of Science Editors , the 6 th edition Publication Manual of the American Psychological Association , and Uniform Requirements for Manuscripts Submitted to Biomedical Journals . These policies are consistent with one another. Relevant aspects are described below.

Conflict of Interest (COI) - Authors are required to declare to the editor any conflicts of interest (any financial or other arrangements or commitments that could reasonably be perceived as sources of bias in the design, interpretation, or reporting of the results of the research) upon acceptance in the journal. If there is something you would be prudent to declare, please do so in your cover letter. If none of the authors have situations that might reasonably be perceived as a COI, please state this in your cover letter. Undisclosed conflicts later identified by a third party will be published in an "Errata" if the editors feel the readers should know about such conflicts.

Protection of Human Subjects - Manuscripts reporting data from human subjects research must state, in the Method section, what formal review and approval or waiver was granted by appropriate research ethics committee(s). The treatment of research participants must be in accord with ethical and other requirements, as set forth in the country in which the research was conducted and as specified by the sponsoring agency.

Informed Consent - The approach(es) and method(s) of obtaining informed consent should be described, including the reasons why any unconventional approaches or waivers were deemed more ethical and respectful in the particular culture and context. This discussion of the informed consent processes and rationales should be more than perfunctory, given that research published in JERHRE derives from a wide range of cultures that may require unconventional approaches to informed consent (e.g., Aboriginal communities), contexts (e.g., health outreach work in migrant communities), and methods (e.g., ethnographic studies of parents and children in pediatric research units). Considerable ethical problem solving often must enter into the development of effective research procedures including the establishment of a respectful relationship with the research participants and others in their community. Indeed, a major aim of JERHRE is to contribute to understanding and solving ethical problems of this nature.

If you or your funder wish your article to be freely available online to nonsubscribers immediately upon publication (gold open access), you can opt for it to be included in Sage Choice, subject to the payment of a publication fee. The manuscript submission and peer review procedure is unchanged. On acceptance of your article, you will be asked to let Sage know directly if you are choosing Sage Choice. To check journal eligibility and the publication fee, please visit Sage Choice . For more information on open access options and compliance at Sage, including self/author archiving deposits (green open access) visit Sage Publishing Policies on our Journal Author Gateway .

At Sage, we are committed to facilitating openness, transparency and reproducibility of research. Where relevant, The Journal encourages authors to share their research data in a suitable public repository subject to ethical considerations and where data is included, to add a data accessibility statement in their manuscript file. Authors should also follow data citation principles. For more information please visit the Sage Author Gateway , which includes information about Sage’s partnership with the data repository Figshare.

• Read Online
• Sample Issues
• Current Issue
• Email Alert
• Permissions
• Foreign rights
• Reprints and sponsorship
• Advertising

Individual Subscription, Combined (Print & E-access)

Individual Subscription, Print Only

Institutional Subscription, E-access

Institutional Subscription, Combined (Print & E-access)

Student Subscription, Print Only

To order single issues of this journal, please contact SAGE Customer Services at 1-800-818-7243 / 1-805-583-9774 with details of the volume and issue you would like to purchase.

• USC Libraries
• Research Guides
• Identify Empirical Articles

*Education: Identify Empirical Articles

• Google Scholar tips and instructions
• Newspaper Articles
• Video Tutorials
• EDUC 508 Library Session
• Statistics and Data
• Tests & Measurements
• Citation Managers
• APA Style This link opens in a new window
• Scan & Deliver and Interlibrary Loan
• Educational Leadership
• Global Executive Ed.D.
• Marriage & Family Therapy
• Organizational Change & Leadership
• Literacy Education
• Accreditation
• Journal Ranking Metrics
• Publishing Your Research
• Education and STEM Databases

How to Recognize Empirical Journal Articles

Definition of an empirical study:  An empirical research article reports the results of a study that uses data derived from actual observation or experimentation. Empirical research articles are examples of primary research.

Parts of a standard empirical research article:  (articles will not necessary use the exact terms listed below.)

• Abstract  ... A paragraph length description of what the study includes.
• Introduction ...Includes a statement of the hypotheses for the research and a review of other research on the topic.
• Who are participants
• Design of the study
• What the participants did
• What measures were used
• Results ...Describes the outcomes of the measures of the study.
• Discussion ...Contains the interpretations and implications of the study.
• References ...Contains citation information on the material cited in the report. (also called bibliography or works cited)

Characteristics of an Empirical Article:

• Empirical articles will include charts, graphs, or statistical analysis.
• Empirical research articles are usually substantial, maybe from 8-30 pages long.
• There is always a bibliography found at the end of the article.

Type of publications that publish empirical studies:

• Empirical research articles are published in scholarly or academic journals
• These journals are also called “peer-reviewed,” or “refereed” publications.

Examples of such publications include:

• American Educational Research Journal
• Computers & Education
• Journal of Educational Psychology

Databases that contain empirical research:  (selected list only)

• List of other useful databases by subject area

This page is adapted from Eric Karkhoff's  Sociology Research Guide: Identify Empirical Articles page (Cal State Fullerton Pollak Library).

Sample Empirical Articles

Roschelle, J., Feng, M., Murphy, R. F., & Mason, C. A. (2016). Online Mathematics Homework Increases Student Achievement. AERA Open .  ( L INK TO ARTICLE )

Lester, J., Yamanaka, A., & Struthers, B. (2016). Gender microaggressions and learning environments: The role of physical space in teaching pedagogy and communication.  Community College Journal of Research and Practice , 40(11), 909-926. ( LINK TO ARTICLE )

• << Previous: Newspaper Articles
• Next: Workshops and Webinars >>
• Last Updated: Mar 12, 2024 4:56 PM
• URL: https://libguides.usc.edu/education

Data and Statistical Sources: Empirical Articles: Finding Empirical Articles

• About Empirical Articles
• Finding Empirical Articles

Search strategies

This page primarily describes how to find empirical articles using the EBSCO databases that the library subscribes to. However, there are many databases at Cornell that are not presented in the EBSCO format. Find them listed by subject at this link . You can use similar strateg to find empirical articles.  

You may also add specific statistical terms to your search, such as chi, t-test, p-value, or standard deviation. Try searching with terms used in the scientific method: method, results, discussion, or conclusion.

Empirical Articles in EBSCO

Cornell subscribes to scores of databases that provide full text journal articles. Many of the databases are purchased from EBSCO and can be searched using its interface.

Here is a sample search in the Business Source Complete database. Other EBSCO databases with empirical articles and a similar search interface are listed in the box called "EBSCO Databases.

Note that "Economics - Statistical Methods" is a subject term . It is combined with the keyword  "labor economics." Instead of typing 'DE "ECONOMICS -- Statistical Methods'' in the search box, you can just type ECONOMICS -- Statistical Methods and select "Subject Term" from the drop down menu.

EBSCO Databases

• Academic Search Premier This multi-disciplinary database provides full text for more than 8,500 journals, including full text for more than 4,600 peer-reviewed titles. PDF backfiles to 1975 or further are available for well over one hundred journals, and searchable cited references are provided for more than 1,000 titles.
• Business Source Complete Business Source Complete provides full text for scholarly business journals and other sources, including full text for more than 1,800 peer-reviewed business publications. Coverage includes virtually all subject areas related to business. This database provides full text (PDF) for top scholarly journals, including the Harvard Business Review. It also includes industry and country reports from Euromonitor and company and industry reports from Datamonitor.
• EconLit with Full Text Abstracts, indexing, and full-text articles in all fields of economics, including capital markets, country studies, econometrics, economic forecasting, environmental economics, government regulations, labor economics, monetary theory, and urban economics.
• PsycINFO Contains citations and summaries of the international literature in psychology and related behavioral and social sciences, including psychiatry, sociology, anthropology, education, pharmacology, and linguistics. Includes applied psychology, communication systems, developmental psychology, educational psychology, experimental human and animal psychology, personality, physical and psychological disorders, physiological psychology and intervention, professional personnel and issues, psychometrics, social processes and issues, sports psychology and leisure, and treatment and prevention.
• Sociology Source Ultimate An expanded version of SocINDEX, including greater coverage of peer-reviewed journals, international resources and open access titles. Provides citations and direct links to the texts of journal articles, book chapters and conference proceedings, some as far back as 1880. Comprehensive coverage encompassing sub-disciplines and related areas of the social sciences, including labor, crime, demography, economic sociology, immigration, ethnic, racial and gender studies, family, political sociology, religion, development, social psychology, social structure, social work, socio-cultural anthropology, social history, theory, methodology, and more.”
• MEDLINE Compiled by the U.S. National Library of Medicine (NLM), MEDLINE is the world's most comprehensive source of life sciences and biomedical bibliographic information. It contains nearly eleven million records from over 7,300 different publications from 1965 to present.

Search Terms

Some keywords for research studies:

• Empirical Studies
• Observations
• Methodology
• Correlation
• Standard Deviation
• << Previous: About Empirical Articles
• Last Updated: Feb 3, 2023 9:13 AM
• URL: https://guides.library.cornell.edu/empiricalarticles

Open access

• Published: 22 December 2022

A systematic review of high impact empirical studies in STEM education

• Yeping Li 1 ,
• Yu Xiao 1 ,
• Ke Wang 2 ,
• Nan Zhang 3 , 4 ,
• Yali Pang 5 ,
• Ruilin Wang 6 ,
• Chunxia Qi 7 ,
• Zhiqiang Yuan 8 ,
• Jianxing Xu 9 ,
• Sandra B. Nite 1 &
• Jon R. Star 10  

International Journal of STEM Education volume  9 , Article number:  72 ( 2022 ) Cite this article

10k Accesses

13 Citations

33 Altmetric

Metrics details

The formation of an academic field is evidenced by many factors, including the growth of relevant research articles and the increasing impact of highly cited publications. Building upon recent scoping reviews of journal publications in STEM education, this study aimed to provide a systematic review of high impact empirical studies in STEM education to gain insights into the development of STEM education research paradigms. Through a search of the Web of Science core database, we identified the top 100 most-cited empirical studies focusing on STEM education that were published in journals from 2000 to 2021 and examined them in terms of various aspects, including the journals where they were published, disciplinary content coverage, research topics and methods, and authorship’s nationality/region and profession. The results show that STEM education continues to gain more exposure and varied disciplinary content with an increasing number of high impact empirical studies published in journals in various STEM disciplines. High impact research articles were mainly authored by researchers in the West, especially the United States, and indicate possible “hot” topics within the broader field of STEM education. Our analysis also revealed the increased participation and contributions from researchers in diverse fields who are working to formulate research agendas in STEM education and the nature of STEM education scholarship.

Introduction

Two recent reviews of research publications, the first examining articles in the International Journal of STEM Education (IJSTEM) and the second looking at an expanded scope of 36 journals, examined how scholarship in science, technology, engineering, and mathematics (STEM) education has developed over the years (Li et al., 2019 , 2020a ). Although these two reviews differed in multiple ways (e.g., the number of journals covered, the time period of article publications, and article selection), they shared the common purpose of providing an overview of the status and trends in STEM education research. The selection of journal publications in these two reviews thus emphasized the coverage and inclusion of all relevant publications but did not consider publication impact. Given that the development of a vibrant field depends not only on the number of research outputs and its growth over the years but also the existence and influence of some high impact research articles, here we aimed to identify and examine those high impact research publications in STEM education in this review.

Learning from existing reviews of STEM education research

Existing reviews of STEM education have provided valuable insights about STEM education scholarship development over the years. In addition to the two reviews mentioned above, there are many other research reviews on different aspects of STEM education. For example, Chomphuphra et al. ( 2019 ) reviewed 56 journal articles published from 2007 to 2017 covering three popular topics: innovation for STEM learning, professional development, and gender gap and career in STEM. They identified and selected these journal articles through searching the Scopus database and two additional journals in STEM education that were not indexed in Scopus at that time. Several other reviews have been conducted and published with a focus on specific topics, such as the assessment of the learning assistant model (Barrasso & Spilios, 2021 ), STEM education in early childhood (Wan et al., 2021 ), and research on individuals' STEM identity (Simpson & Bouhafa, 2020 ). All of these reviews helped in summarizing and synthesizing what we can learn from research on different topics related to STEM education.

Given the on-going rapid expansion of interest in STEM education, the number of research reviews in STEM education research has also been growing rapidly over the years. For example, there were only one or two research reviews published yearly in IJSTEM just a few years ago (Li, 2019 ). However, the situation started to change quickly over the past several years (Li & Xiao, 2022 ). Table 1 provides a summary list of research reviews published in IJSTEM in 2020 and 2021. The journal published a total of five research reviews in 2020 (8%, out of 59 publications), which then increased to seven in 2021 (12%, out of 59 publications).

Taking a closer look at these research reviews, we noticed that three reviews were conducted with a broad perspective to examine research and trends in STEM education (Li et al., 2020a , 2020b ) or STEAM (science, technology, engineering, arts, and mathematics) education (Marin-Marin et al., 2021 ). Relatively large numbers of publications/projects were reviewed in these studies to provide a general overview of research development and trends. The other nine reviews focused on research on specific topics or aspects in STEM education. These results suggest that, with the availability of a rapidly accumulating number of studies in STEM education, researchers have started to go beyond general research trends to examine and summarize research development on specific topics. Moreover, across these 12 reviews, researchers used many different approaches to search multiple data sources (often with specified search terms) to identify and select articles, including journal publications, research reports, conference papers, or dissertations. It appears that researchers have been creative in developing and using specific approaches to select and review publications that are pertinent to their topics. At the same time, however, none of these reviews were designed and conducted to identify and review high impact research articles that had notable influences on the development of STEM education scholarship.

The importance of examining high impact empirical research publications in STEM education

STEM education differs from many other fields, as STEM itself is not a discipline. There are diverse perspectives about the disciplinarity of STEM and STEM education (e.g., Erduran, 2020 ; Li et al., 2020a ; Takeuchi et al., 2020 ; Tytler, 2020 ). The complexity and ambiguity in viewing and examining STEM and STEM education presents challenges as well as opportunities for researchers to explore and specify what and how they do in ways different from and/or connected with traditional education in the individual disciplines of science, technology, engineering, and mathematics.

Although the field of STEM education is still in an early stage of its development, STEM education has experienced tremendous growth over the past decade. This field has evolved from traditional individual discipline-based education in STEM fields to multi- and interdisciplinary education in STEM. The development of STEM education has been supported by multiple factors, including research funding (Li et al., 2020b ) and the growth of research publications (Li et al., 2020a ). High impact publications play a very large role in the growth of the field, as they are read and cited frequently by others and serve to shape the development of scholarship in the field more than other publications.

Among high impact research publications, we can identify several different types of articles, including empirical studies, research reviews, and conceptual or theoretical papers. Research reviews and conceptual/theoretical papers are very valuable, as they synthesize existing research on a specific topic and/or provide new perspective(s) and direction(s), but they are typically not empirical studies. Review articles aim to provide a summary of the current state of the research in the field or on a particular topic, and they help readers to gain an overview about a topic, key issues and publications. Thus, they are more about what has been published in the literature about a topic and less about reporting new empirical evidence about a topic. Similarly, theoretical or conceptual papers tend to draw on existing research to advance theory or propose new perspectives. In contrast, empirical studies require the use and analysis of empirical data to provide empirical evidence. While reporting original research has been typical in empirical studies in education, these studies can also be secondary analyses of empirical data that test hypotheses not considered or addressed in previous studies. Empirical studies are generally published in academic, peer-reviewed journals and consist of distinct sections that reflect the stages in the research process. With the aim to gain insights about research development in STEM education, we thus decided to focus here on empirical studies in STEM education. Examining and reviewing high impact empirical research publications can help provide us a better understanding about emerging trends in STEM education in terms of research topics, methods, and possible directions in the future.

Considerations in identifying and selecting high impact empirical research publications

Publishing as a way of disseminating and sharing knowledge has many types of outlets, including journals, books, and conference proceedings. Different publishing outlets have different advantages in reaching out to readers. Researchers may search different data sources to identify and select publications to review, as indicated in Table 1 . At the same time, journal publications are commonly chosen and viewed as one of the most important outlets valued by the research community for knowledge dissemination and exchange. Specifically, there are two important advantages in terms of evaluating the quality and impact of journal publications over other formats. First, journal publications typically go through a rigorous peer-review process to ensure the quality of manuscripts for publication acceptance based on certain criteria. In educational research, some common criteria being used include “Standards for Reporting on Empirical Social Science Research in AERA Publications” (AERA, 2006 ), “Standards for Reporting on Humanities-Oriented Research in AERA Publications” (AERA, 2009 ), and “Scientific Research in Education” (NRC, 2002 ). Although the peer-review process is also employed in assessing and selecting proposals or papers for publication acceptance in other formats such as books and conference proceedings, the peer-review process employed by journals (esp. those reputable and top journals in a field) tends to be more rigorous and selective than other publication formats. Second, the impact of journals and their publications has frequently been evaluated by peers and different indexing services for inclusion, such as Clarivate’s Social Sciences Citation Index (SSCI) and Elsevier’s Scopus. The citation information collected and evaluated by indexing services provides another important measure about the quality and impact of selected journals and their publications. Based on these considerations, we decided to select and review those journal publications that can be identified as having high citations to gain an overview of their impact on the research development of STEM education.

Focusing on the selection and review of journal publications with high citations has also been used by many other scholars. For example, Martín‐Páez et al. ( 2019 ) conducted a literature review to examine how STEM education is conceptualized, used, and implemented in educational studies. To ensure the quality of published articles for review, they searched and selected journal articles published in the 2013–2018 period from the Web of Science (WoS) database only. Likewise, Akçayır and Akçayır ( 2017 ) conducted a systematic literature review on augmented reality used in educational settings. They used keywords to search all SSCI-indexed journals from WoS database to identify and select published articles, given that WoS provides easy access to search SSCI indexed articles. In addition to the method of searching the WoS database, some researchers used other approaches to identify and select published articles with high citations. For example, some researchers may search different databases to identify and select articles for reviews, such as Scopus (Chomphuphra et al., 2019 ) and Google (Godin et al., 2015 ). In comparison, however, the WoS core database is more selective than many others, including Scopus. The WoS is the world’s leading scientific citation search and analytical information platform (Li et al., 2018 ), and has its own independent and thorough editorial process to ensure journal quality together with the most comprehensive and complete citation network ( https://clarivate.com/webofsciencegroup/solutions/webofscience-ssci/ ). Its core database has been commonly used as a reliable indexing database with close attention to high standard research publications with a peer-review process and is thus used in many research review studies (e.g., Akçayır & Akçayır, 2017 ; Li et al., 2018 ; Marín-Marín et al., 2021 ; Martín‐Páez et al., 2019 ).

It should be noted that some researchers have used a different approach to identify and select high impact publications other than focusing on article citations. This alternative approach is to identify leading journals from specific fields first and then select relevant articles from these journals. For example, Brown ( 2012 ) identified and selected eight important journals in each STEM discipline after consulting with university faculty and K-12 teachers. Once these journals were selected, Brown then located 60 articles that authors self-identified as connected to STEM education from over 1100 articles published between January 1, 2007 and October 1, 2010. However, as there was no well-established journal in STEM education until just a few years ago (Li et al., 2020a ), the approach used by Brown may be less useful for identifying high impact publications in the field of STEM education. In fact, researchers in STEM education have been publishing their high-quality articles in many different journals, especially those well-established journals with an impact factor. Thus, this approach will not help ensure the selection of high impact articles in STEM education, even though they were selected from well-recognized journals rooted in each of STEM disciplines.

In summary, we searched the WoS core database to identify and select high impact empirical research articles in STEM education as those highly cited articles published in journals indexed and collected in the WoS.

Current review

Similar to previous research reviews (e.g., Li et al., 2020a ), we need to specify the scope of the current review with specific considerations of the following two issues:

What time period should be considered?

How should we identify and select highly cited research publications in STEM education?

Time period

As discussed in a previous review (Li et al., 2020a ), the acronym STEM did not exist until the early 2000s. The existence of the acronym has helped to focus attention on and efforts in STEM education. Thus, consistent with the determination of the time period used in the previous review on examining the status and trends in STEM education, we decided to select articles starting from the year 2000. At the same time, we can use the acronym of STEM as an identifier in locating journal articles in a way as done before (Li et al., 2020a ) and also by others (e.g., Brown, 2012 ; Mizell & Brown, 2016 ). We chose the end of 2021 as the end of the time period for publication search and inclusion.

Searching and identifying highly cited empirical research journal publications in STEM education

To identify and select journal articles in STEM education from the WoS core database, we decided to use the common approach of keyword searches as used in many other reviews (e.g., Gladstone & Cimpian, 2021 ; Winterer et al., 2020 ). Li et al. ( 2020a ) also noted the complexity and ambiguity of identifying publications in STEM education. Thus, we planned to identify and select publications in STEM education as those self-identified by authors. As mentioned above, we then used the acronym STEM (or STEAM) as key terms in our search for publications in STEM education.

Different from the previous review on research status and trends in STEM education (Li et al., 2020a ), the current review aimed to identify and select high impact journal articles but not coverage. Thus, we decided to define and limit the scope of high impact empirical research journal publications as the top 100 most-cited empirical research journal publications obtained from the WoS core database.

Research questions

Li et al. ( 2020a ) showed that STEM education articles have been published in many different journals, especially with the limited journal choices available in STEM education. Given a broader range of journals and a longer period of time to be covered in this review, we can thus gain some insights through examining multiple aspects of the top 100 most-cited empirical studies, including journals in which these empirical studies were published, publication years, disciplinary content coverage, research topics and methods. In addition, recent reviews suggested the value of examining possible trends in the authorship and school level focus (Li, 2022 ; Li & Xiao, 2022 ). Taken together, we are interested in addressing the following six research questions:

What are the top 100 most-cited empirical STEM education research journal publications?

What are the distributions and patterns of the top 100 most-cited empirical research publications in different journals?

What is the disciplinary content coverage of the top 100 most-cited empirical research journal publications and possible trends?

What are research topics and methods of the top 100 most-cited empirical research journal publications?

What are the corresponding authors’ nationalities/regions and professions?

What are school level foci of the top 100 most-cited empirical research journal publications over the years?

Based on the above discussion, we carried out the following steps for this systematic review to address these research questions.

Searching and identifying the top 100 most-cited empirical research journal publications in STEM education

Figure  1 provides a summary of the article search and selection process that was used for this review. The process started with a search of the WoS core database on September 12, 2022 under the field of “topic” (covering title, abstract, author keywords, and keywords plus), using the search terms: “STEM” OR “STEAM” OR “science, technology, engineering, and mathematics”. Because there are many different categories in the WoS database, we then specified the publication search using the four WoS categories listed under “education”: “Education Educational Research,” “Education Scientific Disciplines,” “Psychology Educational,” and “Education Special.” The time period of publication search was further specified as starting from 2000 to 2021.

Flowchart of publication search, identification, and selection process

The search returned 9275 publications under “Education Educational Research,” 2161 under “Education Scientific Disciplines,” 247 under “Psychology Educational,” and 15 under “Education Special.” The combined list of all publications was then placed in descending order in terms of citation counts up to the search date of Sept. 12, 2022, and each publication record was screened one-by-one by three researchers using the inclusion or exclusion criteria (see Table 2 ). At times when the publication record listed was not detailed enough, we searched and obtained the full article to screen and check to determine its eligibility. The process ended after identifying and selecting the top 100 most-cited empirical research journal publications.

Data analysis

To address research question 3, we categorized all 100 publications in terms of the number of STEM disciplines covered in a study. Two general categories were used for this review: publications within a single discipline of STEM vs. those with multi- or inter-disciplines of STEM. In contrast to the detailed classifications used in a previous review (Li et al., 2020a ), this simplified classification can help reveal overall trends in disciplinary content coverage and approach reflected in high impact empirical research in STEM education.

To examine research topics, we used the same list of topics from previous reviews (Li & Xiao, 2022 ; Li et al., 2020a ). The following list contains the seven topic categories (TCs) that were used to classify and examine all 100 publications identified and selected from the search in this study.

TC1: Teaching, teacher, and teacher education in STEM (including both pre-service and in-service teacher education) in K-12 education;

TC2: Teacher and teaching in STEM (including faculty development, etc.) at post-secondary level;

TC3: STEM learner, learning, and learning environment in K-12 education;

TC4: STEM learner, learning, and learning environments (excluding pre-service teacher education) at post-secondary level;

TC5: Policy, curriculum, evaluation, and assessment in STEM (including literature reviews about a field in general);

TC6: Culture, social, and gender issues in STEM education;

TC7: History, epistemology, and perspectives about STEM and STEM education.

To examine research methods, we coded all publications in terms of the following methodological categories: (1) qualitative methods, (2) quantitative methods, and (3) mixed methods. We assigned each publication to only one research topic and one method, following the process used in the previous reviews (Li et al., 2019 , 2020a ). When there was more than one topic or method that could have been used for a publication, a final decision was made in choosing and assigning the primary topic and/or method after discussion.

To address research question 5, we examined the corresponding author’s (or the first author, if no specific indication was given about the corresponding author) nationality/region and profession. Many publications in STEM education have joint authorship but may contain limited information about different co-authors. Focusing on the corresponding author’s nationality/region is a feasible approach as we learned from a previous research review (Li et al., 2020a ). For the corresponding author’s profession, we used the same two general categories from the recent reviews (Li, 2022 ; Li & Xiao, 2022 ): “education” and “STEM+” that differentiate a corresponding author’s profession in education/educational research vs. disciplines and fields other than education. If a publication’s corresponding author was listed as affiliated with multiple departments/institutions, the first department/institution affiliation was chosen and used to identify the author’s nationality/region and profession.

To answer research question 6, we adopted the three categories from recent research reviews: K-12, postsecondary, and general (Li, 2022 ; Li & Xiao, 2022 ). The use of these school level categories helped reveal the distribution of STEM education research interests and development over the school level span. While the first two categories are self-explanatory, the “general” category is for those empirical research publications on questions or issues either pertinent to all school levels or that cross the boundary of K-12 school and college.

Results and discussion

The following sections are structured to report findings as corresponding to each of the six research questions.

Top 100 most-cited empirical research articles from 2000 to 2021

Figure  2 shows the distribution of the top 100 most-cited empirical research journal publications in STEM education over the years 2000–2021. As the majority of these publications (72 out of 100, 72%) were published between 2011 and 2016, the results suggest that publications typically need about 5–10 years to accumulate high enough citations for inclusion. Research articles published more than 10 years ago would likely become out-of-dated, unless those studies have been recognized as classic in the field. Some recent publications (6 publications, 2018–2019) emerged with high citations could suggest the emergency of interesting ‘hot’ topics in the field.

Distribution of the top 100 most-cited empirical research publications over the years (Note: all 100 of these most-cited publications were published in the years 2005-2019.)

To have a more fine-grained sense of these highly cited research articles, we took a more detailed look at the top ten most-cited publications from the search (see Table 3 ). These ten most-cited publications were published between 2005 and 2014, with an average of 337 citations and a range of 238–820 citations per article. Only two of the top ten articles were published before 2010; both gained very impressive citations over the years (820 citations for the article published in 2009 and 289 citations for the other published in 2005). The on-going high citations of these two research articles are clear indication of their impact and importance in the field.

Table 3 also shows that the top ten list of most-cited empirical research articles were published in six different journals, with the majority of these journals focusing on general educational research or educational psychology. The importance of STEM education research was clearly recognized with high impact publications in these well-established journals. At the same time, the results imply the rapid development of STEM education research in its early stages and the value of examining possible trends in journals that published high impact articles in STEM education over time.

Moreover, we noticed that all of these top ten articles had corresponding authors who were from the U.S., with the exception of one by researchers in the U.K. This result is consistent with what we learned from previous reviews of STEM education research publications (Li et al., 2019 , 2020a ). About 75% of STEM-related journal publications were typically contributed by U.S. scholars, either in this journal’s publications from 2014 to 2018 (Li et al., 2019 ) or publications from 36 journals from 2000 to 2018 (Li et al., 2020a ). It is not surprising that all of these high impact research publications from 2005 to 2014 were contributed by researchers in the West, especially the United States. (Below we report more about the corresponding authorship of the 100 high impact research publications beyond the top 10 that are reported here.)

Distributions and patterns of highly cited publications across different journals

Forty-five journals were identified as publishing these top 100 most-cited articles. Table 4 shows that the majority (26) of these journals focus on general educational research or educational psychology, publishing 52 of the top 100 most-cited articles. Fourteen journals with titles specifying a single discipline of STEM published 38 of these top 100 articles, three journals with two specified STEM disciplines in their titles published seven of these articles, one journal with three specified STEM disciplines published one article, and one journal specifying all four STEM disciplines published two articles. Among these 45 journals, 36 journals are indexed in SSCI, with the remaining nine journals indexed in ESCI (Emerging Sources Citation Index). These are clearly all reputable and well-established journals, with 36 established before 2000 and 9 established in or after 2000. Only three journals in the list are Open Access (OA) journals, and they were all established after 2000. The results suggest that researchers have been publishing high impact STEM education research articles in a wide range of well-established traditional journals, with the majority in general educational research or educational psychology with a long publishing history. It further confirms that the importance of STEM education research has been well-recognized in educational research or educational psychology as noted above. At the same time, the results imply that the history of STEM education itself has been too brief to establish its own top journals and identity except only one in STEM education (IJSTEM) (Li et al., 2020a ).

Among these 45 journals listed in Table 4 , we classified them into two general categories: general education research journals (26, all without mention of a discipline of STEM in a journal’s title) and those (19) with one or more STEM disciplines specified in a journal’s title. Figure  3 presents the distributions of these top 100 articles in these two general categories over the years. Among 49 articles published before 2014, the majority (31, 63%) of these articles were published in journals on general educational research or educational psychology. However, starting in 2014, a new trend emerged with more of these highly cited articles (30 out of 51, 59%) published in journals with STEM discipline(s) specified. The result suggests a possible shift of developing and gaining disciplinary content consciousness in STEM education research publications.

Trend of the top 100 most-cited articles published in journals without vs. with subject discipline(s) of STEM specified. (Note: 0 = journals without STEM discipline specified, 1 = journals with STEM discipline(s) specified.)

As a further examination of the distribution of publications in journals specified with STEM discipline(s), Fig.  4 shows the distributions of these highly cited articles in different journal categories over the years. It is clear that these highly cited articles were typically published in journals on general educational research or educational psychology before 2014. However, things started to change since 2014, with these highly cited articles published in more diverse journals including those with STEM discipline(s) specified in the journal titles. The journals that include only a single discipline of STEM have been more popular than others among those journals that specify one or more STEM disciplines. The result is not surprising as journals specified with a single discipline of STEM are more common, often with a long publishing history and support from well-established professional societies of education on a single discipline of STEM. This trend suggests that the importance of STEM education has also gained increasing recognition from professional societies that used to focus on a single discipline of STEM.

Distribution of highly cited research articles across different journal categories over the years. (Note: 0 = journals without STEM discipline specified, 1 = journals with a single discipline of STEM specified, 2 = journals with two disciplines of STEM specified, 3/4 = journals with 3 or 4 disciplines of STEM specified.)

To glimpse into those recent changes, we took a closer look at the six articles published in 2018 and 2019 as examples (see Table 5 ). All of these articles have been highly cited in just 3 or 4 years, with an average of 102 citations (range, 75–144) per article. Across these six articles, the majority were published in journals whose titles specified one or more STEM disciplines: three in journals with a single discipline of STEM specified, one in a journal on STEM education, and two in journals on general educational research. At the same time, these recent publications are not specifically on any single discipline of STEM, but multi- and interdisciplinary STEM education.

Disciplinary content coverage

The search of STEM education publications from the WoS core database relied on several keywords that the authors used to self-identify their research on STEM education. After coding and categorizing all top 100 publications, 25 research publications were found as focusing on a single discipline of STEM and 75 publications on multi- and interdisciplinary STEM education. The majority of these 100 most-cited empirical studies, in their focus on multi- and interdisciplinary STEM education, reflects the overall focus in STEM education, a trend consistent with what was learned from a previous review of journal publications in STEM education (Li et al., 2020a ).

Among the 25 research articles on a single discipline of STEM, the majority of these articles (56%, 14 out of 25) focused on science, 5 articles on technology, 4 articles on mathematics, and 2 articles on engineering. The result suggests that of the four STEM disciplines, arguably “science” is the broadest category and so it is not surprising that the number of publications on science is the most prevalent. Indeed, the result is also consistent with what we can learn from Table 4 . Among the 14 journals specifying a single STEM discipline that published 38 of the top 100 articles, seven journals focus on “science” that published 27 of these 38 articles.

To examine possible trends over time, Fig.  5 shows the distribution of these 100 articles across these two disciplinary content coverage categories over the years. For each of the publishing years from 2005 to 2019, there were always more high citation empirical publications on multi- and interdisciplinary STEM education than high citation publications focusing on a single discipline of STEM. Moreover, there were no high citation publications on a single discipline of STEM before 2011 or after 2017 that made the cut for inclusion in the top 100 list. These results suggest an overall trend of on-going emphasis on multi- and interdisciplinary research in STEM education, which can be further verified by what we learned from the six recent publications in Table 5 .

Publication distribution by disciplinary content coverage over the years. (Note: S = single discipline of STEM, M = multiple disciplines of STEM.)

Research topics and methods

Table 6 presents the distribution of all 100 highly cited publications classified in terms of the seven topic categories (TCs) over the years. Overall, all seven TCs have publications that were on the top 100 high citation publication list. There were clearly the most publications on TC6 (culture, social, and gender issues in STEM education), followed by publications on TC4 (STEM learner, learning, and learning environments at post-secondary level). The large number of publications with high citations in these two categories suggest possible evolution of research interests and topics in the field of STEM education. Taking a closer look at the six recent publications in Table 5 , it is clear that culture, social, and gender issues were the focus in these recent publications, with the exception of one publication on assessment. This result presents a picture that appears somewhat different from what we learned from previous research reviews that did not focus exclusively on high impact publications from the WoS database (Li & Xiao, 2022 ; Li et al., 2020a ).

Looking at the distribution of these publications within each of the seven TCs, “culture, social, and gender issues in STEM education” (TC6) is a topic area that consistently has some highly cited research publications in almost each of the publishing years. “STEM learner, learning, and learning environments at post-secondary level” (TC4) also has some consistent and on-going research interest with highly cited publications making the list in most of these publishing years. In contrast, publication distributions in the rest of the TCs did not present clearly notable patterns over the years.

Figure  6 shows the number of publications distributed over the years by research methods in these empirical studies. The use of quantitative methods (71) is dominant overall and is especially prevalent among these most-cited publications in the years from 2005 to 2019, a result consistent with what we learned from a previous research review (Li et al., 2020a ). Across these three methodological classifications, qualitative methods were used in 20 empirical studies, and mixed methods were used in only 9 empirical studies. Comparatively, there were many more articles published between 2010 and 2016 that used quantitative methods than the other two methods. However, there were somehow less dramatic differences in method use among empirical studies published either before 2010 or after 2016. As the use of different methods can help reveal ways of collecting and analyzing data to provide empirical evidence, it would be interesting to learn more about possible development and use of research methods in STEM education in the future as a new empirical research paradigm.

Publication distribution in terms of research methods over the years. (Note: 1 = qualitative, 2 = quantitative, 3 = mixed.)

Corresponding author’s nationality/region and profession Footnote 1

Examining the corresponding author’s nationality/region helps reveal the international diversity in research engagement and scholarly contribution to STEM education. Figure  7 indicates 87 highly cited publications (87%, out of 100 publications) with the corresponding author from the United States, followed by 6 publications (6%) contributed by researchers in the U.K., and the remaining 7 publications with the corresponding author from seven other countries/regions (i.e., one publication for each country/region). The results show some international diversity in terms of the number of country/region represented, but with a clear dominance of research contributions from the West especially the United States. The result echoes what we learned above about the corresponding author’s nationality/region for the top ten most-cited articles (see Table 3 ).

Distribution of corresponding author’s nationality/region of the top 100 articles

Recent reviews of journal publications in IJSTEM suggest a trend of increasing diversity in research contributions from many more different countries/regions (Li, 2022 ; Li & Xiao, 2022 ). We would not be surprised if the list of top 100 most-cited empirical research publications contained more contributions from other countries/regions in the future.

After coding the corresponding author’s profession in these top 100 articles, we found that similar numbers of publications had corresponding authors who were researchers in education (49) and STEM+ (51). This result is consistent with what we learned from the corresponding authors’ profession distribution in recent publications in IJSTEM (Li, 2022 ). The diversity in contributing to STEM education scholarship from researchers with various disciplinary training is evident.

To examine possible trends in the corresponding authors’ profession over time, Fig.  8 shows the distributions of these publications in the two profession categories over the years. It is interesting to note that researchers in education typically served as the corresponding authors for more articles published before 2014: 31 articles by researchers in education and 18 articles by researchers in STEM+ for a total of 49 published before 2014. However, a new trend has emerged since 2014, with many more researchers in STEM+ serving as the corresponding authors for these highly cited research articles: 18 articles by researchers in education and 33 articles by researchers in STEM+ for a total of 51 published since 2014.

Distribution of publications by corresponding author’s profession over the years. (Note: 1 = education, 2 = STEM+)

This trend is consistent with what we learned above about the increased number of these publications in journals specified with STEM discipline(s) since 2014 (see Figs. 3 and 4 ). We see an increasing number of researchers in STEM+ fields contributing and publishing empirical research articles in many journals associated with STEM discipline(s) since 2014, resulted in an increase in citations from professional communities while furthering the development of STEM education scholarship. The result is also consistent with what we learned from the authorship development of publications in IJSTEM over the years (Li & Xiao, 2022 ), an increasing trend of having STEM education scholarship contributions from diverse STEM+ fields.

Publications by school level over the years

With an increasing trend of contributions from researchers in diverse STEM+ fields, the identification of school level can help reveal where these high impact research publications focus on issues in STEM education. The coding results show that the majority (63) of these 100 most-cited articles focused on issues at the postsecondary level, 30 articles on issues at the K-12 school level, and 7 articles in the category of “general.”

Figure  9 presents the distributions of these highly cited publications across these three school categories over the years. It is interesting to note that high impact publications on issues at the postsecondary level outnumbered those in other two categories in almost every of these publishing years. As educational issues in K-12 school level were typically attended to by researchers in education, the increasing number of contributions from researchers in diverse STEM+ fields likely pushed the number of citations on publications that fit their interests more at the postsecondary level. The result is consistent with a growing trend in IJSTEM publications on STEM education at the post-secondary level revealed in a recent review (Li & Xiao, 2022 ).

Distribution of highly cited publications by school level focus and year. (Note: 1 = K-12 school level, 2 = Post-secondary level, 3 = General.)

We also noticed that almost no articles in the category of “general” before 2011 and after 2015 made to the list of top 100 most-cited publications. This result suggests that high impact empirical research in STEM education was conducted more at the school level rather than on issues across the boundary of K-12 school and college. With an increasing number of publications in the “general” category noted in recent review of IJSTEM publications (Li & Xiao, 2022 ), it would be interesting to learn more about cross-school boundary development of STEM education scholarship in the future.

Concluding remarks

This systematic review of high impact empirical studies in STEM education explores the top 100 most-cited research articles from the WoS database as published in journals from 2000 to 2021. These articles were published in a wide range of 45 reputable and well-established journals, typically with a long publishing history. These publications present an overall emphasis more on multi- and interdisciplinary STEM education rather than a single discipline of STEM, with an increasing trend of publishing in journals whose title specified one or more STEM discipline(s). Before 2014, 37% (18 out of 49) of these most-cited articles were published in journals whose title specified with a STEM discipline(s). In contrast, 59% (30 out of 51) articles were published in such journals since 2014, and even more so with 67% of the six articles published in 2018 and 2019. This trend is further elevated with two of those high impact articles recently published in this journal, International Journal of STEM Education . There appears a growing sense of developing disciplinary content consciousness and identity in STEM education.

Consistent with our previous reviews (Li et al., 2019 , 2020a ), the vast majority of these highly cited STEM research publications were contributed by authors from the West, especially the United States where STEM and STEAM education originated. Although there were contributions from eight other countries/regions in these top 100 publications, the diversity of international engagement and contribution was limited. Our results also provide an explanation of what may become “hot” topics among these highly cited articles. In particular, the topic of “culture, social, and gender issues in STEM education” is quite prevalent among those highly cited research publications, followed by the topic area of “STEM learner, learning, and learning environments at post-secondary level.” In comparison, topics related to disciplinary content integration in STEM teaching and learning and STEM teacher training have not yet emerged as “hot” among these highly cited empirical studies. Given that an increasing trend of diversity was noted from a review of recent publications in IJSTEM (Li, 2022 ), we would not be surprised if there will be more high impact research publications contributed by researchers from many other countries/regions on diverse topics in the future.

As STEM education does not have a long history, there will be many challenges and opportunities for new development in STEM education. One important dimension is research method. Among the top 100 most-cited empirical studies, quantitative methods were used as the dominant approach, followed by qualitative methods and then mixed methods. This is not surprising as research in multidisciplinary STEM education may require the use and analysis of data across different disciplines, more frequently in large quantitative data than in other data formats. However, when research questions evolve in the future, it would be interesting to learn more about method development and use in STEM education as a new research paradigm.

We started this review with the intention of gaining insights into the development of STEM education scholarship beyond what we learned about publication growth in STEM education from prior reviews. Indeed, this systematic review provided us with the opportunity to learn about possible trends and gaps in different aspects as discussed above. At the same time, we can learn even more by making connections across these different aspects. One important question in STEM education is to understand the nature of STEM education scholarship and to find ways of developing STEM education scholarship. However, STEM is not a discipline by itself, which suggests possible fundamental differences between STEM education scholarship and scholarship typically defined and classified for a single discipline of STEM. With the increasing participation and contributions from researchers in diverse STEM+ fields as we learned from this review, there is a good possibility that the nature of STEM education scholarship will be collectively formulated with numerous contributions from diverse scholars. Continuing analyses of high impact publications is an important and interesting topic that can yield more insights in the years to come.

Availability of data and materials

The data and materials used and analyzed for the report were obtained through searching the Web of Science database, and related journal information are available directly from these journals’ websites.

Our analysis found that the vast majority (94%) of these top 100 articles had the same researcher to serve as the first author and the corresponding author. There are 10 articles that had more than one corresponding authors, and we chose the first corresponding author as listed in our coding.

Abbreviations

Association for computing machinery  AERA

American Educational Research Association

Cell biology education

Emerging Sources Citation Index

Institute of electrical and electronics engineers

International Journal of STEM Education

Kindergarten-Grade 12

National Research Council 

Social Sciences Citation Index

Science, technology, engineering, and mathematics

Disciplines or fields other than education, including those commonly considered under the STEM umbrella plus some others

Science, technology, engineering, arts, and mathematics

Topic category

Web of Science

Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20 , 1–11.

Article   Google Scholar  

American Educational Research Association (AERA). (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35 (6), 33–40.

American Educational Research Association (AERA). (2009). Standards for reporting on humanities-oriented research in AERA publications. Educational Researcher, 38 (6), 481–486.

Barrasso, A. P., & Spilios, K. E. (2021). A scoping review of literature assessing the impact of the learning assistant model. International Journal of STEM Education, 8 , 12. https://doi.org/10.1186/s40594-020-00267-8

Brown, J. (2012). The current status of STEM education research. Journal of STEM Education: Innovations & Research, 13 (5), 7–11.

Google Scholar  

Chomphuphra, P., Chaipidech, P., & Yuenyong, C. (2019). Trends and research issues of STEM education: A review of academic publications from 2007 to 2017. Journal of Physics: Conference Series, 1340 (2019), 012069.

Erduran, S. (2020). Nature of “STEM”? Science & Education, 29 , 781–784. https://doi.org/10.1007/s11191-020-00150-6

Gao, X., Li, P., Shen, J., & Sun, H. (2020). Reviewing assessment of student learning in interdisciplinary STEM education. International Journal of STEM Education, 7 , 24. https://doi.org/10.1186/s40594-020-00225-4

Gladstone, J. R., & Cimpian, A. (2021). Which role models are effective for which students? A systematic review and four recommendations for maximizing the effectiveness of role models in STEM. International Journal of STEM Education, 8 , 59. https://doi.org/10.1186/s40594-021-00315-x

Godin, K., Stapleton, J., Kirkpatrick, S. I., Rhona, M., Hanning, R. M., & Leatherdale, S. T. (2015). Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews, 4 , 138. https://doi.org/10.1186/s13643-015-0125-0

Jackson, C., Mohr-Schroeder, M. J., Bush, S. B., Maiorca, C., Roberts, T., Yost, C., & Fowler, A. (2021). Equity- Oriented Conceptual Framework for K-12 STEM literacy. International Journal of STEM Education, 8 , 38. https://doi.org/10.1186/s40594-021-00294-z

Li, K., Rollins, J., & Yan, E. (2018). Web of Science use in published research and review papers 1997–2017: A selective, dynamic, cross-domain, content-based analysis. Scientometrics, 115 (1), 1–20.

Li, Y. (2019). Five years of development in pursuing excellence in quality and global impact to become the first journal in STEM education covered in SSCI. International Journal of STEM Education, 6 , 42. https://doi.org/10.1186/s40594-019-0198-8

Li, Y. (2022). Eight years of development in welcoming and engaging diverse scholars to share and promote STEM education research worldwide. International Journal of STEM Education, 9 , 69. https://doi.org/10.1186/s40594-022-00385-5

Li, Y., Froyd, J. E., & Wang, K. (2019). Learning about research and readership development in STEM education: A systematic analysis of the journal’s publications from 2014 to 2018. International Journal of STEM Education, 6 , 19. https://doi.org/10.1186/s40594-019-0176-1

Li, Y., Wang, K., Xiao, Y., & Froyd, J. E. (2020a). Research and trends in STEM education: A systematic review of journal publications. International Journal of STEM Education, 7 , 11. https://doi.org/10.1186/s40594-020-00207-6

Li, Y., Wang, K., Xiao, Y., Froyd, J. E., & Nite, S. B. (2020b). Research and trends in STEM education: A systematic analysis of publicly funded projects. International Journal of STEM Education, 7 , 17. https://doi.org/10.1186/s40594-020-00213-8

Li, Y., & Xiao, Y. (2022). Authorship and topic trends in STEM education research. International Journal of STEM Education, 9 , 62. https://doi.org/10.1186/s40594-022-00378-4

Marín-Marín, J.-A., Moreno-Guerrero, A.-J., Dúo-Terrón, P., & López-Belmonte, J. (2021). STEAM in education: A bibliometric analysis of performance and co-words in Web of Science. International Journal of STEM Education, 8 , 41. https://doi.org/10.1186/s40594-021-00296-x

Martín-Páez, T., Aguilera, D., Perales-Palacios, F. J., & Vílchez-González, J. M. (2019). What are we talking about when we talk about STEM education? A review of literature. Science Education, 103 , 799–822.

Mizell, S., & Brown, S. (2016). The current status of STEM education research 2013–2015. Journal of STEM Education: Innovations & Research, 17 (4), 52–56.

National Research Council (NRC). (2002). Scientific research in education . Committee on Scientific Principles for Education Research. Shavelson, R. J., and Towne, L., Editors. Center for Education. Division of Behavioral and Social Sciences and Education. National Academy Press.

Nguyen, K. A., Borrego, M., Finelli, C. J., DeMonbrun, M., Crockett, C., Tharayil, S., Shekhar, P., Waters, C., & Rosenberg, R. (2021). Instructor strategies to aid implementation of active learning: a systematic literature review. International Journal of STEM Education , 8 , 9. https://doi.org/10.1186/s40594-021-00270-7

Reinholz, D. L., White, I. & Andrews, T. (2021). Change theory in STEM higher education: a systematic review. International Journal of STEM Education , 8 , 37. https://doi.org/10.1186/s40594-021-00291-2

Simpson, A., & Bouhafa, Y. (2020). Youths’ and adults’ identity in STEM: A systematic literature review. Journal for STEM Education Research, 3 , 167–194.

Takeuchi, M. A., Sengupta, P., Shanahan, M.-C., Adams, J. D., & Hachem, M. (2020). Transdisciplinarity in STEM education: A critical review. Studies in Science Education, 56 (2), 213–253.

Tytler, R. (2020). STEM education for the twenty-first century. In J. Anderson & Y. Li (Eds.), Integrated approaches to STEM education: An international perspective (pp. 21–43). Springer.

Chapter   Google Scholar  

Wahono, B., Lin, P. L. & Chang, C. Y. (2020). Evidence of STEM enactment effectiveness in Asian student learning outcomes. International Journal of STEM Education , 7 , 36. https://doi.org/10.1186/s40594-020-00236-1

Wan, Z. H., Jiang, Y., & Zhan, Y. (2021). STEM education in early childhood: A review of empirical studies. Early Education and Development, 32 (7), 940–962. https://doi.org/10.1080/10409289.2020.1814986

Winterer, E. R., Froyd, J. E., Borrego, M., Martin, J. P., & Foster, M. (2020). Factors influencing the academic success of Latinx students matriculating at 2-year and transferring to 4-year US institutions—implications for STEM majors: A systematic review of the literature. International Journal of STEM Education, 7 , 34. https://doi.org/10.1186/s40594-020-00215-6

Zhao, F. & Schuchardt, A. (2021). Development of the Sci-math Sensemaking Framework: categorizing sensemaking of mathematical equations in science. International Journal of STEM Education , 8 , 10. https://doi.org/10.1186/s40594-020-00264-x

Download references

Acknowledgements

The author would like to thank Marius Jung and the staff at SpringerOpen for their support in publishing this article.

This work was supported by National Social Science Foundation of China, BHA180134.

Author information

Authors and affiliations.

Texas A&M University, College Station, TX, 77843, USA

Yeping Li, Yu Xiao & Sandra B. Nite

Nicholls State University, Thibodaux, USA

Tianjin Normal University, Tianjin, China

Southwest University, Chongqing, China

Shanghai Normal University, Shanghai, China

Capital Normal University, Beijing, China

Ruilin Wang

Beijing Normal University, Beijing, China

Hunan Normal University, Changsha, China

Zhiqiang Yuan

Yangzhou University, Yangzhou, China

Jianxing Xu

Harvard University, Cambridge, USA

Jon R. Star

You can also search for this author in PubMed   Google Scholar

Contributions

YL conceived the study, helped with article search and screening, conducted data analyses, and drafted the manuscript. YX and KW contributed with article search, identification, selection and coding. NZ, YP, RW, CQ, ZY, and JX contributed with data coding. SBN and JRS reviewed drafts and contributed to manuscript revisions. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yeping Li .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Li, Y., Xiao, Y., Wang, K. et al. A systematic review of high impact empirical studies in STEM education. IJ STEM Ed 9 , 72 (2022). https://doi.org/10.1186/s40594-022-00389-1

Download citation

Received : 27 November 2022

Accepted : 01 December 2022

Published : 22 December 2022

DOI : https://doi.org/10.1186/s40594-022-00389-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Empirical studies
• Research review
• STEM education scholarship

• Ask a Librarian

Research: Overview & Approaches

• Getting Started with Undergraduate Research
• Planning & Getting Started
• Building Your Knowledge Base
• Locating Sources
• Reading Scholarly Articles
• Creating a Literature Review
• Productivity & Organizing Research
• Scholarly and Professional Relationships

Introduction to Empirical Research

Databases for finding empirical research, guided search, google scholar, examples of empirical research, sources and further reading.

• Interpretive Research
• Action-Based Research
• Creative & Experimental Approaches

Your Librarian

• Introductory Video This video covers what empirical research is, what kinds of questions and methods empirical researchers use, and some tips for finding empirical research articles in your discipline.

• Guided Search: Finding Empirical Research Articles This is a hands-on tutorial that will allow you to use your own search terms to find resources.

• Study on radiation transfer in human skin for cosmetics
• Long-Term Mobile Phone Use and the Risk of Vestibular Schwannoma: A Danish Nationwide Cohort Study
• Emissions Impacts and Benefits of Plug-In Hybrid Electric Vehicles and Vehicle-to-Grid Services
• Review of design considerations and technological challenges for successful development and deployment of plug-in hybrid electric vehicles
• Endocrine disrupters and human health: could oestrogenic chemicals in body care cosmetics adversely affect breast cancer incidence in women?

• << Previous: Scholarly and Professional Relationships
• Next: Interpretive Research >>
• Last Updated: Apr 5, 2024 9:55 AM
• URL: https://guides.lib.purdue.edu/research_approaches

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

The Neuroscience of Growth Mindset and Intrinsic Motivation

Our actions can be triggered by intentions, incentives or intrinsic values. Recent neuroscientific research has yielded some results about the growth mindset and intrinsic motivation. With the advances in neuroscience and motivational studies, there is a global need to utilize this information to inform educational practice and research. Yet, little is known about the neuroscientific interplay between growth mindset and intrinsic motivation. This paper attempts to draw on the theories of growth mindset and intrinsic motivation, together with contemporary ideas in neuroscience, outline the potential for neuroscientific research in education. It aims to shed light on the relationship between growth mindset and intrinsic motivation in terms of supporting a growth mindset to facilitate intrinsic motivation through neural responses. Recent empirical research from the educational neuroscience perspective that provides insights into the interplay between growth mindset and intrinsic motivation will also be discussed.

1. Introduction

With an emphasis on inquiry and scientific skills, students are encouraged to discover, produce and evaluate knowledge, using inquiry and scientific skills [ 1 ]. Such inquiry learning should be structured in a way that student learning is facilitated, while encouraging students to plan and conduct their own investigation. An autonomy-supportive environment facilitates autonomous learning, and fosters self-determined motivation in students [ 2 ]. Students learn to synthesize contradictory perspectives and rise to intellectual meta-levels of thinking, which is a crucial trait for the 21st-century operating environment [ 3 ]. As such, it is fundamental to nurture the young generation in becoming adaptive, self-regulated and self-determined.

In the 21st century, there has been a strong proliferation of research on growth mindset and intrinsic motivation in learning. The constructs of mindset and motivation have been important foci among educators seeking to positively impact student learning and outcomes. The underlying mechanism for students to have their own agency in finding out new knowledge is intrinsic motivation. However, much of this research has relied on quantitative approaches for assessing students’ self-reports on motivational regulations and learning outcomes [ 4 , 5 ]. Some of these quantitative findings are used to generalize across school settings. Although the multiple facets of student motivation and learning have been identified in quantitative analyses, they have not provided a detailed understanding of students’ motivational processes. Neuroscience methods may offer new insights regarding students’ motivation and learning processes.

Most neuroscience studies have focused on research related to cognitive functions, such as attention, memory and decision-making. In addition to these cognitive studies, there is also the implicit nature of mindsets that lead to the malleability of self-attributes (e.g., intelligence) [ 6 ]. Subtle feedback and messages related to growth mindset can have noticeable effects on students’ attitudes and motivation that may transfer to long-term outcomes. Likewise, human motivation is important, as it is one’s intrinsic desire to learn and obtain information. Growth mindset is the belief that intelligence can be nurtured through learning and effort, while intrinsic motivation is the volition to engage in a task for inherent satisfaction. Individuals with growth mindset believe that motivation can be nurtured, and that extrinsic motivation can be internalized (i.e., from extrinsic regulation to integrated regulation that is similar to intrinsically motivated behavior). In an integrative view, growth mindset and intrinsic motivation are important and interrelated, thus raising fundamental questions about the neural mechanisms of mindset-motivation interaction. The links among growth mindset, brain and motivation are important to academic performance. Therefore, it is important to draw on neuroscientific findings to show the way the brain is motivated, and how it learns by changing mindset (i.e., from a fixed to a growth mindset). Such intervention studies are still not common, and there is potential in these research areas.

This paper reviews the theoretical frameworks of growth mindset and intrinsic motivation, and how they are linked to neuroscientific evidence. It also reviews a number of recent neuroscience studies related to growth mindset and intrinsic motivation. It is important to survey the progress of neuroscience research on growth mindset and intrinsic motivation, as understanding the neural substrates will provide insights into human motivation and drive. Neuroscience research has the potential to support and refine models of motivation and cognitive skill. It may play a pivotal role in developing classroom interventions and understanding non-cognitive skills (e.g., mindset). Knowing the key brain regions that are associated with growth mindset and intrinsic motivation, researchers and practitioners could work together to investigate the granular processes of motivation in relation to growth mindset.

Most empirical research on growth mindset and intrinsic motivation has focused on behavioral methods and self-reports of experiences. There is little information about the internal processes of motivation at a higher level of resolution. It is, therefore, relevant and timely to examine the existing literature and empirical research that is associated with intrinsic motivation. Neuroscientific evidence has the potential to uncover new insights and refine the conceptual ideas of intrinsic motivation by articulating the granular processes of motivation that behavioral methods alone cannot afford. This paper offers recommendations for potential neuroscience research in studying growth mindset and intrinsic motivation.

2. Growth Mindset

Growth mindset is defined as a belief that construes intelligence as malleable and improvable [ 6 ]. Students with growth mindset are likely to learn by a mastery approach, embrace challenges and put in effort to learn. For instance, growth-minded individuals perceive task setbacks as a necessary part of the learning process and they “bounce back” by increasing their motivational effort [ 7 , 8 ]. One recent study on elementary students showed that leveraging an online educational game (the BrainPOP website) with in-game rewards can promote a growth mindset by directly incentivizing effort and encouraging persistence in low performing students [ 7 ]. Learners with growth mindset tend to embrace lifelong learning and the joy of incremental personal growth. In addition, they do not see their intelligence or personality as fixed traits. They will mobilize their learning resources without being defeated by the threat of failure. This paper aims to provide some insights into the cultivation of resilience and mastery in university students, preparing them to overcome challenges in the real working world.

Empirical studies have revealed that growth mindset has positive effects on student motivation and academic performance [ 9 , 10 ]. Recent research has also shown that mindset is related to student outcomes and behaviors including academic achievement, engagement, and willingness to attempt new challenges [ 11 , 12 ]. Numerous studies have shown the effects of growth mindset interventions on students’ achievement at all ages. According to Dweck [ 9 ], teaching growth mindset to junior high school students resulted in increased motivation and better academic achievement. Her findings revealed that students in the growth mindset intervention group outperformed those in the control group (who received excellent training in study skills), indicating improved learning and desire to work hard. The growth mindset intervention teaches students that intelligence is not a fixed quality [ 13 ]. Intelligence can be nurtured through challenging tasks, as intelligence grows with hard work on challenging problems. A growth mindset intervention was especially impactful with student outcomes in particular subjects such as science and mathematics [ 14 ].

An individual with a growth mindset works hard and improves without an incentive reward in mind as the outcome. The conceptualization of growth mindset is similar to that of intrinsic motivation. A learner with a growth mindset tends to self-regulate their own learning and has the propensity to cope with academic tasks. Hence, encouraging a growth mindset can improve the academic performance of college students [ 14 , 15 ] and middle school math students [ 9 ].

Most of the abovementioned empirical studies reported the utility of questionnaires or self-report measures. There is still limited neuroscientific research on the neural mechanism of growth mindset. It is, therefore, important to examine data from other means such as neuroscientific information about how the brain changes with experience of learning and how it is associated to growth mindset. The subsequent sections will discuss the neuroscientific evidence of growth mindset.

3. Intrinsic Motivation

Intrinsic motivation is inherent, as it drives the direction of an individual’s behavior and self-determination [ 16 ]. Self-determination is important in the development of beings to become more effective and refined in their reflection of ongoing experiences [ 17 ]. When students experience the inherent satisfaction of the activity itself, they will show intrinsically motivated behavior. If students are doing the activity in order to attain some reward, such as grades or social recognition, they are extrinsically motivated [ 18 ]. Students’ motivated behaviors pertaining to choice, effort and persistence in academic tasks correspond directly with their level of intrinsic motivation [ 19 , 20 ].

Numerous studies have examined the effects of intrinsic motivation, including the adaptive consequences for individuals such as exposing them to novel situations and developing their diverse competencies to cope with unforeseen circumstances [ 21 ]. In addition, intrinsic motivation is the propensity for individuals to learn about new subjects and to differentiate their interests, thereby fostering a sense of purpose and meaning [ 22 ]. Recent empirical findings have shown that intrinsic motivation is a key factor in academic achievement [ 23 ] and pursuit of interest [ 24 ], thus fostering learning and growth.

Dopamine is the predominant neurotransmitter in the brain that aids in controlling the brain’s reward and pleasure centers, as well as motivated and emotional behaviors [ 25 ]. Dopamine neurons that are excited by unexpected reward events project to the striatum, cortex, limbic system and hypothalamus, thus affecting physiological functions and motivated behaviors. Dopamine is considered a key substrate of intrinsic motivation, thus promoting attentiveness and behavioral engagement [ 25 ]. For instance, participants were likely to voluntarily engage with the task during a free-choice time period [ 26 ] or a self-determined choice condition [ 27 ]. These consistent findings indicate that an enhanced activity within the dopaminergic value system whereby perceived autonomy support promotes intrinsic motivation. As such, learning is a neural process that requires the reinforcement of synaptic functioning and is strongly mediated by dopamine and attentional gain in the frontal cortex [ 28 ]. Positive and negative affect will also strengthen or weaken the learner’s intrinsic motivation in a particular subject, thus influencing the attitude towards that subject.

Over the past few decades, behavioral evidence has established the importance of intrinsic motivation and how it impacts one’s learning. However, our understanding of the underlying mechanism of intrinsic motivation is still in its infancy, and it is unclear how one’s intrinsic motivation progresses or changes over time. More evidence is needed to establish the mechanism of intrinsic motivation at a granular level. The recommendation is to include neuroscientific evidence to track and understand which aspects of one’s learning progress determine intrinsic motivation, complementing the existing behavioral evidence. An approach of the neuroscience method is to foster intrinsically motivated behaviors based on task complexity in various contexts, thereby addressing intrinsic motivation through different forms of exploration. The following sections will discuss in detail the neuroscience methods and neuroscientific evidence of intrinsic motivation.

4. Neuroscience Methods

The main neuroscience methods that have been applied in motivation studies are electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). Such neuroscientific research is still considered novel, as most motivational studies have focused on behavioral methods. Both neuroscientific techniques are non-invasive procedures for measuring brain activity. The key difference is fMRI has a higher spatial resolution than EEG, whereas EEG has a better temporal resolution than fMRI.

Neuroscience methods (e.g., fMRI) could provide insights into neural substrates of growth mindset and intrinsic motivation. We could measure the learner’s brain activity and neural responses to a specific task in relation to internal processes of motivation. For instance, intrinsic motivation could be assessed by an experimental task or free-choice behavior measures.

The use of neuroscientific techniques enables us to focus on the learning process rather than the learning outcomes [ 29 ]. The neuroimaging findings offer an understanding of the brain, indicating the specific areas of brain activation which could in turn correlate with the behavioral results. As such, neuroimaging findings might support the self-reported data and explore brain regions with neural activation in relation to changes in performance during an online activity.

5. Neural Correlates of Growth Mindset and Intrinsic Motivation

There is a small body of existing growth mindset studies using neuroscience methods. The study by Moser et al. [ 30 ] suggested that individuals with a growth mindset are receptive to corrective feedback, exhibiting a higher Pe (error positivity) waveform response, which is correlated with a heightened awareness of and attention to mistakes. Enhanced Pe amplitude was associated with enhanced attention to corrective feedback following errors and subsequent error correction. Individuals with growth mindset are likely to have heightened awareness of and attention to errors. In addition, growth-minded individuals may neutralize the affective response to negative feedback, which could be indicated by neural activation. Anterior cingulate cortex (ACC) is the region of frontal midline cortex that is related to learning and control [ 31 ]. A recent study [ 32 ] found that growth mindset was related to both ventral and dorsal striatal connectivity with dorsal ACC. Dorsal ACC and dorsolateral prefrontal cortex (DLPFC) are critical to error-monitoring and behavioral adaptation. Growth mindset was strongly associated with dorsal and ventral striatal connectivity, as well as DLPFC. Learners with growth mindset are efficient in error-monitoring and receptive to corrective feedback. Hence, growth mindset has the potential to encourage intrinsically motivated behaviors in schools and promote lifelong learning.

Neuroscientific evidence has shown that ACC is associated to cognitive control and motivation [ 31 ]. Neural correlates revealed that dopamine is critical for motivation and cognitive control, with motivation-cognition interactions between midbrain regions and lateral frontal cortex [ 33 ]. Cognitive control is influenced by reward motivation. Participants were assigned to three levels of cognitive controls (low, mid and high). Different beneficial effects of reward (high versus low) were exhibited. Participants with high versus low reward anticipation showed increased activity in the medial and lateral frontal cortex. Brain activity was also stronger at the low level of cognitive control than mid and high levels. These findings demonstrated that motivation plays an important role in the cognitive control. In addition, high-level control tasks may demand an enhancing effect of motivation.

A recent EEG study [ 34 ] showed that school children with growth mindset endorsement performed with higher accuracy after mistakes (i.e., post-error accuracy). The event-related potential (ERP), which is a measure of brain response due to the result of error and correct trials, revealed that Pe amplitude difference was largest at site Pz (i.e., midline parietal). Together with the behavioral data, correlational analyses showed that having a higher growth mindset was associated with a larger Pe difference. Students with attentional resources are able to remember their mistakes and able to make sense of their mistakes, thus correcting themselves during the learning process. Students do not like to take risks that show their weaknesses, such as making mistakes [ 35 ]. However, with growth mindset endorsement, students are not afraid to make mistakes, as they have the ability to learn with post-error accuracy. Hence, growth-minded students will be resilient and self-regulated when faced with obstacles or challenges during their learning process.

Little is known about the interplay between neural responses and intrinsic motivation. Intrinsically motivated action can be characterized by an individual’s engagement in behavior for one’s own sake, with free-choice time on a task [ 36 ]. An empirical study measured intrinsic motivation by examining a network of brain regions as the participants spent free-choice time on a word problem task [ 37 ]. Using fMRI, a network of brain regions revealed diminished task-related activity, predicting subsequent increased intrinsic motivation. The neuroimaging data suggest that decreased activation of neural cognitive control is associated with increased intrinsic motivation, thus extending one’s task engagement. Another recent study by Lee and Reeve [ 38 ] examined the neural substrates of intrinsic motivation during task performance. Their findings showed activated anterior insular cortex (AIC; a limbic-related cortex region) when students performed intrinsically motivated tasks. These neural findings are consistent with the concept of intrinsic motivation in terms of pursuit and interest satisfaction as intrinsic rewards. Based on these findings, it was concluded that AIC activity and its functional interactions are linked to an intrinsic-motivation neural system [ 38 ].

Two recent motivation studies used free-choice measures, such as a stop-watch (SW) game, as an experimental task to assess participants’ intrinsic motivation [ 39 , 40 ]. A traditional SW game includes a stopwatch that starts automatically, and the player tries to stop the watch at a specific time. Experimental stimuli were presented on the computer screen and participants were required to use the keypad to complete the SW tasks. It is interesting to note the relationship between the optimal challenge condition and intrinsic motivation using EEG [ 39 ]. Students performed better when they felt optimally challenged, and had enhanced intrinsic motivation in the game experiment. Stimulus-preceding negativity (SPN) is considered to be an electrophysiological indicator of motivation level. The EEG findings showed a larger SPN during the feedback anticipation period of the near miss condition than in the complete defeat condition, suggesting that participants were more intrinsically motivated to win in close games [ 39 ]. For the second study, fMRI was used to explore the degree of enjoyment for the preference levels of SW game [ 40 ]. It was found that participants had enhanced intrinsic motivation when they played the SW game with the action-outcome contingency condition. The fMRI findings revealed significant activation in the regions of the mid brain and ventral striatum in the action-outcome contingency condition, indicating that the intrinsic value of an action and achieving success. These two studies suggest that neuroscience methods are used to assess individuals’ intrinsic motivation using a free-choice experiment, such as a SW game. However, using game elements and design may have implications for authentic learning programs. Using the game approach, students may have enhanced intrinsic motivation for doing the activities in a gaming format or platform. Adopting the game approach and translating such motivation-enhancing elements into classrooms may seem challenging and time-consuming. Such experimental tasks are usually carried out in a closed environment, such as in a controlled laboratory setting within the fMRI facility.

Intrinsic motivation is associated with sensitivity of feedback processing in the striatum [ 41 ]. The striatum plays a key role in reinforcing learning as it receives input from midbrain dopamine neurons and produces adaptive behaviors. Striatum activity is associated with reward processing, indicating that an intrinsically motivated task could foster the individual’s intrinsic motivation. For instance, feedback-related responses in the striatum can potentially promote or undermine intrinsic motivation of a desired behavior. Positive feedback was viewed as a rewarding outcome, and highly motivated subjects could attune to the feedback despite of fatigue through the study [ 41 ]. Performance-feedback may have affective salient response to striatum and produce a motivated behavior. A study by Lee [ 42 ] showed that intrinsic motivation was related to the AIC that is known to be associated with the sense of agency, while extrinsic motivation was associated with posterior parietal regions (e.g., posterior cingulate cortex, angular gyrus). The type of task also plays a very important role in activating the AIC. Lee [ 42 ] also found that interesting tasks activated the AIC and ventral striatum (i.e., brain region for reward processing), but not uninteresting tasks. AIC relates to the satisfaction of intrinsic need, whereas ventral striatum relates to the feeling of reward. His findings suggest that AIC and ventral striatum activations are associated with intrinsic motivation.

Intrinsic motivation is difficult to measure in an objective manner. In order to track one’s intrinsic motivation, it requires one to perform an experimental task over time. For instance, one’s brain activity can be tracked during the process of performing an intrinsically motivated or optimally challenged task. Together with behavioral measures, contemporary methods such as fMRI can be used to track the changes in intrinsic motivation during a free-choice activity.

6. The Neuroscientific Interplay between Growth Mindset and Intrinsic Motivation

Based on the abovementioned empirical findings, there is a distinctive neuroscientific interplay between growth mindset and intrinsic motivation. EEG findings could not directly show the brain regions that are related to mindset and motivation. Compared to the EEG, which is based on brain waveforms, fMRI is a better method for showing insights into the brain regions that are associated with growth mindset and intrinsic motivation. It is interesting to note that growth mindset is mainly associated with the dorsal regions of the brain, whereas intrinsic motivation is associated with the mid-brain regions. The common brain areas that are related to both growth mindset and intrinsic motivation are ACC and ventral striatum. Knowing the behavioral correlates for these two brain regions, potential research could investigate the neural correlates of growth mindset and intrinsic motivation. This brings us a step closer to understand the neural mechanism between growth mindset and intrinsic motivation. Below is a table that highlights the neuroscientific evidence of growth mindset and intrinsic motivation in relation to cognition. The behavioral correlate for the brain region is included in parentheses (see Table 1 ).

Neuroscientific evidence of growth mindset and intrinsic motivation.

Growth mindset relates to brain processes, and brain processes relate to motivated behaviors. Likewise, motivated behaviors can affect cognition as motivation shapes what and how people think [ 43 ]. As such, individuals’ goals and needs may be exemplified when they steer their thinking towards desired outcomes. Research has shown that growth mindset has an impact on children’s behavior, particularly in terms of effort, motivation and resilience [ 12 , 44 ]. By understanding the underlying mechanism of intrinsic motivation, teachers are able to guide students in applying the relevant self-regulatory strategies at school. When individuals have intrinsic motivation for performing a task at work or school, their work or educational performance will improve [ 45 , 46 ]. With the inculcation of growth mindset, individuals will perceive the intrinsic value of a given task and self-regulate their behaviors to perform the task. Through internalization, individuals will generate intrinsically motivated behaviors at work or school.

As our brain is plastic, it is able to undergo reorganization and development. Brain plasticity or neuroplasticity refers to the ability of our brain to change throughout our life. It is thereby important to understand how our brain changes if we undergo growth mindset intervention and whether there are changes in our intrinsic motivation as well. This phenomenon is yet to be explored in educational research. It is thus an avenue worth pursuing for educators who hope to make the best of their students with regard to learning and personal growth. Such educational neuroscience research may impact teaching and learning, thus providing a better understanding of the neuroscientific interplay between growth mindset and intrinsic motivation. Future educational neuroscience research may include classroom interventions such as a growth mindset induction and how it affects the neuroscience of intrinsic motivation.

7. Future Directions

The principal intent of this paper is to highlight a potential educational neuroscience research in areas of growth mindset and intrinsic motivation. Although there are some empirical studies on mindset and motivation, the neuroscience of intrinsic motivation is still unclear and at its infancy. There are also limited neuroscientific studies on students’ motivation and learning. As educational neuroscience research looks promising in the near future, we should be aware of the potential integration between neuroscience methods and behavioral measures. For successful intervention studies, there are some considerations that need to be warranted.

First, educators should design a task that has intrinsic value for students to be engaged in doing. For instance, an interesting task will instill curiosity into students, when compared to an uninteresting one. Inculcating the value of doing the task or task value will definitely stimulate the students’ interest. Second, teachers should provide the autonomy or choice for students. Autonomy or the agency of learning is the key substrate to intrinsic motivation [ 17 ]. Research has shown that autonomy is the strongest predictor of intrinsic motivation [ 47 ]. Autonomy is considered the self-endorsement of actions, whereby individuals feel less coerced and they generate autonomous behavior at work or school. In the same vein, choice is the opportunity for individuals to decide and exert control over the situation. A recent study found that the provision of choice, however trivial or inconsequential, might also increase an individual’s intrinsic motivation [ 48 ]. The researchers used behavioral and electrophysiological (i.e., electroencephalogram) evidence to explain the importance of need satisfaction for autonomy to enhance one’s intrinsic motivation toward the task.

Third and finally, performance-related feedback could influence intrinsic motivation [ 41 ]. Participants are likely to perceive their performance on the task differently based on the type of performance-related feedback. For instance, positive feedback may enhance one’s intrinsic motivation, while negative feedback may undermine one’s motivation. In addition, the frequency of performance-related feedback may affect one’s neural processing (i.e., posterior cingulate cortex) in supporting task performance. There were enhanced activity of posterior cingulate cortex and performance gains after the performance-feedback manipulation. This shows that posterior cingulate cortex might facilitate the learning of a task [ 41 ]. The current level of a learner’s intrinsic motivation may also influence the way he or she processes the performance-related feedback. It is still not fully clear how the nature of performance-feedback could affect an individual’s feedback processing. Perhaps neuroscience methods could provide some insights into this area of research.

Based on the neuroscientific evidence, there is an undermining effect of monetary reward on intrinsic motivation; that is, one’s intrinsic motivation is undermined when extrinsic reward is no longer promised [ 26 ]. Neuroscience findings suggest that there are connections between the striatum and the prefrontal cortex in determining the outcome; decreased activation of the striatum and midbrain when the subjects do not get the task value, as well as decreased activation of the lateral prefrontal cortex (LPFC) when they are not motivated to show cognitive engagement with the task. Since growth mindset is a belief system that favors hard work and performance monitoring [ 32 ], a learner’s subjective belief in determining the outcome may modulate activity of the striatum, in response to cognitive feedback that nurtures growth mindset. Hence, neuroscientific evidence may provide insights into the learning and motivational processes that could be helpful for teachers and practitioners in improving their learning and teaching practices, thus supporting student learning and motivation.

8. Concluding Remarks

This paper reviewed the recent empirical neuroscientific studies on growth mindset and intrinsic motivation. Research in these areas is still in its infancy. This paper attempted to provide an overview of the underlying mechanism between growth mindset and intrinsic motivation. Educating students about growth mindset and how they can improve their learning experience is a step toward increased intrinsic motivation in our society. From a personal perspective, intrinsic motivation is the key substrate to learning and development. The promotion of a growth mindset can nurture individuals to learn as they understand that intelligence is malleable. It is important that, as teachers, we show our students the value and importance of learning at schools. With a growth mindset, students will learn with a positive attitude, and they will identify the importance of the contents. Teachers should also embrace a growth mindset such that they will understand the importance of providing autonomy over student learning to enhance self-regulation. As such, students will be more motivated to learn subjects at school, rather than relying on the presumption that students will be interested in learning. This preliminary review paper offers a useful road map for identifying the areas that need to be addressed in neuroscientific research related to growth mindset and intrinsic motivation. However, this paper did not discuss the potential roles of socio-demographic variables and personality traits on growth mindset and intrinsic motivation. Future research will benefit from the continued development of neuroscientific evidence to connect the substantial behavioral evidence of these variables and traits associated with growth mindset and intrinsic motivation.

Acknowledgments

Thanks to all reviewers who contributed to improving the manuscript.

Conflicts of Interest

The author declares no conflict of interest.

EDU 610: Educational Research: Empirical Research

• Library Catalog
• Tips for GALILEO
• Check for Full Text
• Citation Help
• Web Resources
• Learning Commons This link opens in a new window
• Empirical Research
• Postgraduate Research
• Microsoft Word
• Word - Basic Formatting
• Google Docs
• Google - Basic Formatting
• TU Writing Resources
• APA Sample Paper
• APA Citations
• Reference Examples
• Book with Editor(s)
• Chapters and Parts of Books
• Company Reports
• Journal Article
• Magazine Article
• Patents & Laws
• Unpublished Manuscripts/Informal Publications (i.e. course packets and dissertations)
• Installing Zotero
• Gathering Citations with Zotero
• Word Processing with Zotero
• Troubleshooting

[email protected]

229-227-6959

Make an appointment with a Librarian  

Make an appointment with a Tutor 

Follow us on:

Empirical Research Studies

• Defining Empirical Articles
• Searching for Empirical Articles
• Examples of Empirical Articles

What is an empirical article?

Empirical research articles are scholarly, peer-reviewed journal articles that are based on data collected during the authors' real-life experiments or observations. They are primary research documents that contain either qualitative or quantitative research methods:

• Qualitative research focuses on collecting in-depth information from small sample sizes in order to describe a trend. Data is typically collected through one-on-one interviews with participants.
• Quantitative research uses large, representative sample sizes to collect a variety of statistics that can then be generalized. Data is typically collected through a questionnaire, attitude scale, or achievement test.

How can I tell if an article is empirical?

Several factors can help you decide whether or not an article is empirical:

• Academic journals such as Review of Educational Research , Journal of Educational Psychology , and Child Development often publish empirical articles.
• Popular magazines such as Time or Newsweek don't publish empirical articles.
• Professional journals such as the Journal of Technology and Teacher Education or Educational Leadership will publish empirical articles, while professional magazines (e.g., TEACH , Education Today , and Education Matters ) won't publish empirical articles.
• Did the author(s):
• Administer a survey or questionnaire ?
• Conduct an interview ?
• Collect data ?
• Use an assessment to measure results?
• Empirical articles include many of the above characteristics.
• Introduction/Literature Review
• This section will include information on how the study was conducted: how it was designed, who the participants were and how they participated, and how the results were measured.
• This information allows other researchers to replicate the study with their own participants.
• Results/Findings
• Discussion/Conclusion/Implications
• Although authors might combine some sections, label them differently, or not use any headings at all, empirical articles will contain all of the above information.
• Because empirical articles contain so many details about their studies, they tend to be longer.
• They also contain charts, tables, and other graphics to help display the data that was collected.

College of Southern Maryland. CSM Library. (2018, Jan. 19). Empirical research article. Retrieved July 15, 2019, from https://libguides.csmd.edu/empirical_research

Pan, M. L. (2016). Preparing literature reviews: Qualitative and quantitative approaches (5th ed.). New York, NY: Routledge.

Penn State University Libraries. (2019, May 9). Empirical research in the social sciences and education. Retrieved July 15, 2019, from https://guides.libraries.psu.edu/emp

University of La Verne. Wilson Library. (2018, June 26). Identify empirical research articles. Retrieved July 15, 2019, from https://laverne.libguides.com/empirical-articles

GALILEO (and EBSCOhost databases)

Consider adding one of the following key terms to your search:

• Data Analysis
• Field Study
• Investigation
• Observation
• Questionnaire
• Statistical Analysis

For best results, use GALILEO's Advanced Search option. This will let you put your chosen key term in its own search box using the AND operator.

Another option is to limit your results by type:

• Look for the Refine Results column on the left-hand side of your results list.
• Underneath the Limit by Type section, click on Reports .
• Underneath the Limit To section, click on both the Full Text and Scholarly (Peer Reviewed) Journals options.

Look for similar options when searching subject-specific databases.

Education Database (ProQuest)

• Click on the Document type section on the left-hand side of your results list.
• In the drop-down menu that appears, click on the More option.
• Statistics/Data Report

Education Database (ProQuest) http://www.galileo.usg.edu/express?link=zued&inst=thco

ERIC (https://eric.ed.gov/)

• Look for the Descriptor section on the left-hand side of your results list.
• Questionnaires
• Qualitative Research
• Click on the More option to view more descriptors.

• Look for the Publication Type section on the left-hand side of your results list.
• Reports - Research
• Reports - Evaluative
• Tests/Questionnaires
• Numerical/Qualitative Data
• Click on the More option to view more publication types.

Lawrence, A. C., Al-Bataineh, A. T., & Hatch, D. (2018). Educator perspectives on the instructional effects of one-to-one computing implementation. Contemporary Educational Technology, 9 (2), 206-224. https://doi.org/10.30935/cet.414950

Semerci, A. (2018). Students' views on the use of tablet computers in education. World Journal on Educational Technology: Current Issues, 10 (2), 104-114. https://doi.org/10.18844/wjet.v10i2.3420

Turner, K. (2019). One-to-one learning and self-determination theory. International Journal of Instruction, 12 (2), 1-16. https://doi.org/10.29333/iji.2019.1221a

USU Libraries. (2019, April 22). What's empirical research? [Video file]. Retrieved from https://www.youtube.com/watch?v=fZ-LGZdqWLU

• << Previous: Learning Commons
• Next: Postgraduate Research >>
• Last Updated: Oct 23, 2023 1:26 PM
• URL: https://libguides.thomasu.edu/edu610

• University of Memphis Libraries
• Research Guides

Empirical Research: Defining, Identifying, & Finding

Searching for empirical research.

• Defining Empirical Research
• Introduction

Where Do I Find Empirical Research?

How do i find more empirical research in my search.

• Database Tools
• Search Terms
• Image Descriptions

Because empirical research refers to the method of investigation rather than a method of publication, it can be published in a number of places. In many disciplines empirical research is most commonly published in scholarly, peer-reviewed journals . Putting empirical research through the peer review process helps ensure that the research is high quality. 

Finding Peer-Reviewed Articles

You can find peer-reviewed articles in a general web search along with a lot of other types of sources. However, these specialized tools are more likely to find peer-reviewed articles:

• Library databases
• Academic search engines such as Google Scholar

Common Types of Articles That Are Not Empirical

However, just finding an article in a peer-reviewed journal is not enough to say it is empirical, since not all the articles in a peer-reviewed journal will be empirical research or even peer reviewed. Knowing how to quickly identify some types non-empirical research articles in peer-reviewed journals can help speed up your search. 

• Peer-reviewed articles that systematically discuss and propose abstract concepts and methods for a field without primary data collection.
• Example: Grosser, K. & Moon, J. (2019). CSR and feminist organization studies: Towards an integrated theorization for the analysis of gender issues .
• Peer-reviewed articles that systematically describe, summarize, and often categorize and evaluate previous research on a topic without collecting new data.
• Example: Heuer, S. & Willer, R. (2020). How is quality of life assessed in people with dementia? A systematic literature review and a primer for speech-language pathologists .
• Note: empirical research articles will have a literature review section as part of the Introduction , but in an empirical research article the literature review exists to give context to the empirical research, which is the primary focus of the article. In a literature review article, the literature review is the focus. 
• While these articles are not empirical, they are often a great source of information on previous empirical research on a topic with citations to find that research.
• Non-peer-reviewed articles where the authors discuss their thoughts on a particular topic without data collection and a systematic method. There are a few differences between these types of articles.
• Written by the editors or guest editors of the journal. 
• Example:  Naples, N. A., Mauldin, L., & Dillaway, H. (2018). From the guest editors: Gender, disability, and intersectionality .
• Written by guest authors. The journal may have a non-peer-reviewed process for authors to submit these articles, and the editors of the journal may invite authors to write opinion articles.
• Example: García, J. J.-L., & Sharif, M. Z. (2015). Black lives matter: A commentary on racism and public health . 
• Written by the readers of a journal, often in response to an article previously-published in the journal.
• Example: Nathan, M. (2013). Letters: Perceived discrimination and racial/ethnic disparities in youth problem behaviors . 
• Non-peer-reviewed articles that describe and evaluate books, products, services, and other things the audience of the journal would be interested in. 
• Example: Robinson, R. & Green, J. M. (2020). Book review: Microaggressions and traumatic stress: Theory, research, and clinical treatment .

Even once you know how to recognize empirical research and where it is published, it would be nice to improve your search results so that more empirical research shows up for your topic.

There are two major ways to find the empirical research in a database search:

• Use built-in database tools to limit results to empirical research.
• Include search terms that help identify empirical research.
• << Previous: Discussion
• Next: Database Tools >>
• Last Updated: Apr 2, 2024 11:25 AM
• URL: https://libguides.memphis.edu/empirical-research

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 24, Issue 2
• Five tips for developing useful literature summary tables for writing review articles
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0003-0157-5319 Ahtisham Younas 1 , 2 ,
• http://orcid.org/0000-0002-7839-8130 Parveen Ali 3 , 4
• 1 Memorial University of Newfoundland , St John's , Newfoundland , Canada
• 2 Swat College of Nursing , Pakistan
• 3 School of Nursing and Midwifery , University of Sheffield , Sheffield , South Yorkshire , UK
• 4 Sheffield University Interpersonal Violence Research Group , Sheffield University , Sheffield , UK
• Correspondence to Ahtisham Younas, Memorial University of Newfoundland, St John's, NL A1C 5C4, Canada; ay6133{at}mun.ca

https://doi.org/10.1136/ebnurs-2021-103417

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Literature reviews offer a critical synthesis of empirical and theoretical literature to assess the strength of evidence, develop guidelines for practice and policymaking, and identify areas for future research. 1 It is often essential and usually the first task in any research endeavour, particularly in masters or doctoral level education. For effective data extraction and rigorous synthesis in reviews, the use of literature summary tables is of utmost importance. A literature summary table provides a synopsis of an included article. It succinctly presents its purpose, methods, findings and other relevant information pertinent to the review. The aim of developing these literature summary tables is to provide the reader with the information at one glance. Since there are multiple types of reviews (eg, systematic, integrative, scoping, critical and mixed methods) with distinct purposes and techniques, 2 there could be various approaches for developing literature summary tables making it a complex task specialty for the novice researchers or reviewers. Here, we offer five tips for authors of the review articles, relevant to all types of reviews, for creating useful and relevant literature summary tables. We also provide examples from our published reviews to illustrate how useful literature summary tables can be developed and what sort of information should be provided.

Tip 1: provide detailed information about frameworks and methods

• Download figure
• Open in new tab
• Download powerpoint

Tabular literature summaries from a scoping review. Source: Rasheed et al . 3

The provision of information about conceptual and theoretical frameworks and methods is useful for several reasons. First, in quantitative (reviews synthesising the results of quantitative studies) and mixed reviews (reviews synthesising the results of both qualitative and quantitative studies to address a mixed review question), it allows the readers to assess the congruence of the core findings and methods with the adapted framework and tested assumptions. In qualitative reviews (reviews synthesising results of qualitative studies), this information is beneficial for readers to recognise the underlying philosophical and paradigmatic stance of the authors of the included articles. For example, imagine the authors of an article, included in a review, used phenomenological inquiry for their research. In that case, the review authors and the readers of the review need to know what kind of (transcendental or hermeneutic) philosophical stance guided the inquiry. Review authors should, therefore, include the philosophical stance in their literature summary for the particular article. Second, information about frameworks and methods enables review authors and readers to judge the quality of the research, which allows for discerning the strengths and limitations of the article. For example, if authors of an included article intended to develop a new scale and test its psychometric properties. To achieve this aim, they used a convenience sample of 150 participants and performed exploratory (EFA) and confirmatory factor analysis (CFA) on the same sample. Such an approach would indicate a flawed methodology because EFA and CFA should not be conducted on the same sample. The review authors must include this information in their summary table. Omitting this information from a summary could lead to the inclusion of a flawed article in the review, thereby jeopardising the review’s rigour.

Tip 2: include strengths and limitations for each article

Critical appraisal of individual articles included in a review is crucial for increasing the rigour of the review. Despite using various templates for critical appraisal, authors often do not provide detailed information about each reviewed article’s strengths and limitations. Merely noting the quality score based on standardised critical appraisal templates is not adequate because the readers should be able to identify the reasons for assigning a weak or moderate rating. Many recent critical appraisal checklists (eg, Mixed Methods Appraisal Tool) discourage review authors from assigning a quality score and recommend noting the main strengths and limitations of included studies. It is also vital that methodological and conceptual limitations and strengths of the articles included in the review are provided because not all review articles include empirical research papers. Rather some review synthesises the theoretical aspects of articles. Providing information about conceptual limitations is also important for readers to judge the quality of foundations of the research. For example, if you included a mixed-methods study in the review, reporting the methodological and conceptual limitations about ‘integration’ is critical for evaluating the study’s strength. Suppose the authors only collected qualitative and quantitative data and did not state the intent and timing of integration. In that case, the strength of the study is weak. Integration only occurred at the levels of data collection. However, integration may not have occurred at the analysis, interpretation and reporting levels.

Tip 3: write conceptual contribution of each reviewed article

While reading and evaluating review papers, we have observed that many review authors only provide core results of the article included in a review and do not explain the conceptual contribution offered by the included article. We refer to conceptual contribution as a description of how the article’s key results contribute towards the development of potential codes, themes or subthemes, or emerging patterns that are reported as the review findings. For example, the authors of a review article noted that one of the research articles included in their review demonstrated the usefulness of case studies and reflective logs as strategies for fostering compassion in nursing students. The conceptual contribution of this research article could be that experiential learning is one way to teach compassion to nursing students, as supported by case studies and reflective logs. This conceptual contribution of the article should be mentioned in the literature summary table. Delineating each reviewed article’s conceptual contribution is particularly beneficial in qualitative reviews, mixed-methods reviews, and critical reviews that often focus on developing models and describing or explaining various phenomena. Figure 2 offers an example of a literature summary table. 4

Tabular literature summaries from a critical review. Source: Younas and Maddigan. 4

Tip 4: compose potential themes from each article during summary writing

While developing literature summary tables, many authors use themes or subthemes reported in the given articles as the key results of their own review. Such an approach prevents the review authors from understanding the article’s conceptual contribution, developing rigorous synthesis and drawing reasonable interpretations of results from an individual article. Ultimately, it affects the generation of novel review findings. For example, one of the articles about women’s healthcare-seeking behaviours in developing countries reported a theme ‘social-cultural determinants of health as precursors of delays’. Instead of using this theme as one of the review findings, the reviewers should read and interpret beyond the given description in an article, compare and contrast themes, findings from one article with findings and themes from another article to find similarities and differences and to understand and explain bigger picture for their readers. Therefore, while developing literature summary tables, think twice before using the predeveloped themes. Including your themes in the summary tables (see figure 1 ) demonstrates to the readers that a robust method of data extraction and synthesis has been followed.

Tip 5: create your personalised template for literature summaries

Often templates are available for data extraction and development of literature summary tables. The available templates may be in the form of a table, chart or a structured framework that extracts some essential information about every article. The commonly used information may include authors, purpose, methods, key results and quality scores. While extracting all relevant information is important, such templates should be tailored to meet the needs of the individuals’ review. For example, for a review about the effectiveness of healthcare interventions, a literature summary table must include information about the intervention, its type, content timing, duration, setting, effectiveness, negative consequences, and receivers and implementers’ experiences of its usage. Similarly, literature summary tables for articles included in a meta-synthesis must include information about the participants’ characteristics, research context and conceptual contribution of each reviewed article so as to help the reader make an informed decision about the usefulness or lack of usefulness of the individual article in the review and the whole review.

In conclusion, narrative or systematic reviews are almost always conducted as a part of any educational project (thesis or dissertation) or academic or clinical research. Literature reviews are the foundation of research on a given topic. Robust and high-quality reviews play an instrumental role in guiding research, practice and policymaking. However, the quality of reviews is also contingent on rigorous data extraction and synthesis, which require developing literature summaries. We have outlined five tips that could enhance the quality of the data extraction and synthesis process by developing useful literature summaries.

• Aromataris E ,
• Rasheed SP ,

Twitter @Ahtisham04, @parveenazamali

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient consent for publication Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

• Open access
• Published: 10 January 2024

A scoping review of theories, models and frameworks used or proposed to evaluate knowledge mobilization strategies

• Saliha Ziam   ORCID: orcid.org/0000-0002-8892-9572 1 ,
• Sèverine Lanoue 2 ,
• Esther McSween-Cadieux 2 ,
• Mathieu-Joël Gervais 3 ,
• Julie Lane 2 , 4 ,
• Dina Gaid 5 ,
• Laura Justine Chouinard 1 ,
• Christian Dagenais 6 ,
• Valéry Ridde 7 , 8 ,
• Emmanuelle Jean 9 ,
• France Charles Fleury 10 ,
• Quan Nha Hong 5 &
• Ollivier Prigent 2  

Health Research Policy and Systems volume  22 , Article number:  8 ( 2024 ) Cite this article

2351 Accesses

6 Altmetric

Metrics details

Evaluating knowledge mobilization strategies (KMb) presents challenges for organizations seeking to understand their impact to improve KMb effectiveness. Moreover, the large number of theories, models, and frameworks (TMFs) available can be confusing for users. Therefore, the purpose of this scoping review was to identify and describe the characteristics of TMFs that have been used or proposed in the literature to evaluate KMb strategies.

A scoping review methodology was used. Articles were identified through searches in electronic databases, previous reviews and reference lists of included articles. Titles, abstracts and full texts were screened in duplicate. Data were charted using a piloted data charting form. Data extracted included study characteristics, KMb characteristics, and TMFs used or proposed for KMb evaluation. An adapted version of Nilsen (Implement Sci 10:53, 2015) taxonomy and the Expert Recommendations for Implementing Change (ERIC) taxonomy (Powell et al. in Implement Sci 10:21, 2015) guided data synthesis.

Of the 4763 search results, 505 were retrieved, and 88 articles were eligible for review. These consisted of 40 theoretical articles (45.5%), 44 empirical studies (50.0%) and four protocols (4.5%). The majority were published after 2010 ( n  = 70, 79.5%) and were health related ( n  = 71, 80.7%). Half of the studied KMb strategies were implemented in only four countries: Canada, Australia, the United States and the United Kingdom ( n  = 42, 47.7%). One-third used existing TMFs ( n  = 28, 31.8%). According to the adapted Nilsen taxonomy, process models ( n  = 34, 38.6%) and evaluation frameworks ( n  = 28, 31.8%) were the two most frequent types of TMFs used or proposed to evaluate KMb. According to the ERIC taxonomy, activities to “train and educate stakeholders” ( n  = 46, 52.3%) were the most common, followed by activities to “develop stakeholder interrelationships” ( n  = 23, 26.1%). Analysis of the TMFs identified revealed relevant factors of interest for the evaluation of KMb strategies, classified into four dimensions: context, process, effects and impacts.

Conclusions

This scoping review provides an overview of the many KMb TMFs used or proposed. The results provide insight into potential dimensions and components to be considered when assessing KMb strategies.

Peer Review reports

Contribution to the literature

The evaluation of KMb strategies is a critical dimension of the KMb process that is still poorly documented and warrants researchers’ attention.

Our review identified the most common theories, models and frameworks (TMFs) proposed or used to assess KMb strategies and the main components to consider when evaluating a KMb strategy.

By developing an integrative reference framework, this work contributes to improving organizations’ capacity to evaluate their KMb initiatives.

It is widely recognized that research evidence has the potential to inform, guide, and improve practices, decisions, and policies [ 1 ]. Unfortunately, for diverse reasons, the best available evidence is still too seldom taken into account and used [ 2 , 3 , 4 , 5 , 6 , 7 ]. The field of research on knowledge mobilization (KMb) has been growing rapidly since the early 2000s [ 2 , 3 , 8 , 9 , 10 , 11 ]. Its purpose is to better understand how to effectively promote and support evidence use.

Knowledge mobilization is one of many terms and concepts developed over recent decades to describe processes, strategies, and actions to bridge the gap between research and practice. Other common terms often paired interchangeably with the term “knowledge” are “translation”, “transfer”, “exchange”, “sharing” and “dissemination”, among others. [ 12 , 13 ]. Some are more closely linked than others to specific fields or jurisdictions. For this study, we adopted the term knowledge mobilization (KMb) because it conveys the notions of complexity and multidirectional exchanges that characterize research-to-action processes. We used it as an umbrella concept that encompasses the efforts made to translate knowledge into concrete actions and beneficial impacts on populations [ 1 ]. Moreover, the term KMb is also used by research funding agencies in Canada to emphasize the medium- and long-term effects that research knowledge or research results can have on potential users [ 1 , 14 ].

KMb represents all processes from knowledge creation to action and includes all strategies implemented to facilitate these processes [ 14 ]. A KMb strategy is understood as a coordinated set of activities to support evidence use, such as dissemination activities to reach target audiences (for example, educational materials, practical guides, decision support tools) or activities to facilitate knowledge application in a specific context and support professional behaviour change (for example, community of practice, educational meetings, audits and feedback, reminders, deliberative dialogues) [ 15 ]. A KMb process may vary in intensity, complexity or actor engagement depending on the nature of the research knowledge and the needs and preferences of evidence users [ 7 ].

KMb is considered a complex process, in that numerous factors can facilitate or hinder its implementation and subsequent evidence use. The past two decades have seen the emergence of a deeper understanding of these factors [ 2 , 3 , 16 ]. These may be related to the knowledge mobilized (for example, relevance, reliability, clarity, costs), the individuals involved in the KMb process (for example, openness to change, values, time available, resources), the KMB strategies (for example, fit with stakeholder needs and preferences, regular interactions, trust relationships, timing), and organizational and political contexts (for example, culture of evidence use, leadership, resources) [ 2 , 6 , 17 , 18 ]. However, more studies are needed to understand which factors are more important in which contexts, and to evaluate the effects of KMb strategies.

On this last point, while essential, it is often very complex to study KMb impacts empirically to demonstrate the effectiveness of KMb strategies [ 19 , 20 , 21 ]. Partly for this reason, high-quality studies that evaluate process, mechanisms and effects of KMb strategies are still relatively rare [ 2 , 22 , 23 , 24 , 25 ]. As a result, knowledge about the effectiveness of different KMb strategies remains limited [ 10 , 17 , 19 , 23 , 26 , 27 , 28 ] and their development cannot be totally evidence informed [ 3 , 19 , 20 , 23 , 29 , 30 ], which may seem incompatible with the core values and principles of KMb.

The growing interest in KMb has led to an impressive proliferation of conceptual propositions, such as theories, models and frameworks (TMF) [ 2 , 3 , 9 , 11 , 12 , 31 , 32 ]. Many deplore the fact that these are poorly used [ 11 , 30 , 33 ] and insist on the need to test, refine and integrate existing ones [ 3 , 31 , 34 ]. Indeed, the conceptual and theoretical development of the field has outpaced its empirical development. This proliferation appears to have created confusion among certain users, such as organizations that need to evaluate their KMb strategies. Besides implementing and funding KMb strategies, knowledge organizations such as granting agencies, governments and public organizations, universities and health authorities are often required to demonstrate the impact of their strategies [ 21 , 35 , 36 ]. Yet this can be a significant challenge [ 20 , 23 , 29 ]. They may have difficulty knowing which TMFs to choose, in what context and how to use them effectively in their evaluation process [ 12 , 37 ].

Indeed, the evaluation of KMb strategies is still relatively poorly documented, with respect to the phases of their development and implementation. Our aim in this scoping review is to clarify, conceptually and methodologically, this crucial dimension of the KMb process. This would help organizations gain access to evidence-based, operational and easy-to-use evaluation toolkits for assessing the impacts of their KMb strategies.

To survey the available knowledge on evaluation practices for KMb strategies, we conducted a scoping review. According to Munn et al. [ 38 ], a scoping review is indicated to identify the types of available evidence and knowledge gaps, to clarify concepts in the literature and to identify key characteristics or factors related to a concept. This review methodology also allows for the inclusion of a diversity of publications, regardless of their nature or research design, to produce the most comprehensive evidence mapping possible [ 39 ]. The objective of the scoping review was to identify and describe the characteristics of theories, models and frameworks (TMFs) used or proposed to evaluate KMb strategies. The specific research questions were:

What TMFs to evaluate KMb strategies exist in the literature?

What KMb strategies do they evaluate (that is types of KMb objectives, activities, target audiences)?

What dimensions and components are included in these TMFs?

This scoping review was conducted based on the five steps outlined by Arksey and O’Malley [ 39 ]: (1) formulating the research questions; (2) identifying relevant studies; (3) selecting relevant studies; (4) extracting and charting data; and (5) analysing, collating, summarizing and presenting the data. Throughout the process, researchers and knowledge users (KMb practitioners) were involved in decisions regarding the research question, search strategy, selection criteria for studies and categories for data charting. We followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines [ 40 ]. No protocol was registered for this review.

Search strategy and information sources

The search strategy was developed, piloted and refined in consultation with our team’s librarian. Search terms included controlled vocabulary and keywords related to three main concepts: (1) knowledge mobilization (for example [knowledge or evidence or research] and transfer, translation, diffusion, dissemination, mobilization, implementation science, exchange, sharing, use, uptake, evidence-based practice, research-based evidence), (2) evaluation (for example, evaluat*, measur*, impact, outcome, assess, apprais*, indicator) and (3) TMF (for example, framework*, model*, method*, guide*, theor*). See Additional file 1 for the search terms and strategies used in the electronic searches.

The following databases were searched from January 2000 to August 2023: MEDLINE (Ovid), PsycInfo (Ovid), ERIC (ProQuest), Sociological Abstracts (ProQuest), Dissertations & Theses (Proquest), Érudit and Cairn. These databases were chosen to identify relevant references in the health, education and social fields. Several search strategies were tested by the librarian to optimize the retrieval of citations known to the investigators and to increase the likelihood that all relevant studies would be retrieved. We also searched reference lists of included articles and previous systematic reviews [ 11 , 12 , 15 , 41 ].

Eligibility criteria

A publication was considered eligible if it (1) presented or used a theory, model, or framework (TMF), (2) described dimensions or specific components to consider in the evaluation of KMb strategies, (3) presented or discussed KMb strategies or activities (any initiatives to improve evidence use), and (4) proposed outcomes that might result directly or indirectly from the KMb strategies. Studies were excluded from analysis if they (1) presented a TMF to assess the impact of research without mentioning KMb strategies or an intervention not related to KMb and (2) presented evaluation dimensions or components that could not be generalized. We considered publications in English or French. All types of articles and study designs were eligible, including study protocols.

Study selection

The results of the literature search were imported into Covidence, which the review team used for screening. After duplicate articles were removed, the titles and abstracts were screened independently by two of the three reviewers (EMC, MJG, GL). Publications identified as potentially relevant were retrieved in full text and screened independently by three reviewers (EMC, MJG, GL). Discrepancies regarding the inclusion of any publication were resolved through discussion and consensus among reviewers. The principal investigator (SZ) validated the final selection of articles.

Data synthesis

A data charting form was developed in Microsoft Excel and piloted by the research team. Data extracted included study characteristics (authors, authors’ country of affiliation, year, journal, discipline, article type, study setting, study aim), KMb strategies of interest, KMb objectives, KMb target audiences and TMFs used or proposed for KMb evaluation (existing or new TMF, specific dimensions or components of TMF and so on). Data were extracted by a single reviewer (SL, JC or OP) and validated by a second reviewer (SZ). Disagreements were discussed between reviewers and resolved by consensus. No quality appraisal of included studies was conducted, as this is optional in scoping reviews and the purpose was only to describe the content of identified TMFs [ 42 ].

Data analysis and presentation of results

Data were summarized according to study characteristics, KMb strategy characteristics (activities, objectives, target audiences), types of TMFs, and dimensions or components to consider for KMb evaluation. Disagreements during the process were discussed and resolved through consensus (SL, DG, SZ). A KMb strategy might have one or more objectives and include one or more activities. Thus, the objectives and activities of the KMb strategies extracted from the selected studies were summarized based on existing categorizations. The categorization of KMb objectives was inspired by Gervais et al. [ 15 ] and Farkas et al. [ 43 ] (Table  1 ).

The KMb activities were categorized according to the Expert Recommendations for Implementing Change (ERIC) taxonomy [ 44 ]. The activities were first classified according to the full taxonomy and then grouped into the nine categories proposed by Waltz et al. [ 45 ] (Table  2 ).

The TMFs were categorized according to the categories of theoretical approaches described by Nilsen [ 32 ]: process models, evaluation frameworks, determinant frameworks and classic theories (Table  3 ). The category “implementation theories” originally described by Nilsen [ 32 ] was not used because we did not identify any article that fit this category. We also added a category named “logic models” due to the nature of the identified TMFs. Logic models are often used in theory-driven evaluation approaches and are usually developed to show the links among inputs (resources), activities and outputs (outcomes and short-, medium- and long-term effects) [ 46 ].

Finally, the content extracted from the TMFs was analysed using mainly an inductive method. This method allows, among other things, to develop a reference framework or a model from the emerging categories that are evident in the text data [ 50 ].

The classification of concepts is the result of multiple readings and interpretations. The concepts associated with each dimension of the framework were classified according to their meaning. Similar concepts were grouped together to form components. These grouped components were then associated with the subdimensions and main dimensions of the framework.

Search results

The searches yielded 4763 articles. Of those, 4258 were excluded during the title and abstract screening. Of the 505 full-text articles, we retained 88 in our final sample. The results of the search and selection processes (PRISMA flowchart) are summarized in Fig.  1 .

PRISMA flowchart summarizing search strategy and selection results [ 40 ]

Publication characteristics

Most articles were published after 2010 ( n  = 70, 79.5%), with an average of 5 articles per year between 2010 and 2023 compared with an average of 2.1 articles per year between 2001 and 2009; there were no eligible articles from 2000. The search was conducted in August 2023, and only five articles were published in these 7 months of the year. Table 4 presents the main characteristics of the selected articles. A full list of the included articles with their main characteristics is presented in Additional file 2 .

The number of theoretical and empirical articles was relatively similar. Among the theoretical articles, 19 descriptive articles (21.6%) were aimed at describing a KMb strategy, a KMb infrastructure or a TMF related to a specific programme or context; 18 articles (20.5%) synthesized knowledge to propose a TMF (new or revised); and three articles conducted systematic reviews (3.4%).

The empirical articles category included studies with different methodological approaches (quantitative, qualitative, mixed methods). We will not report the details of the methodologies used, as this would result in a long list with few occurrences. The empirical articles can be divided into three categories: (1) studies that evaluated a TMF related to KMb ( n  = 16, 18.2%), (2) studies that evaluated a KMb strategy ( n  = 21, 23.9%) and (3) studies that evaluated both a KMb strategy and a TMF ( n  = 7, 8.0%).

Most articles were related to healthcare ( n  = 71, 80.7%). This field of study was divided into three subdomains. The healthcare and social services articles usually described or assessed a KMb strategy targeting health professionals’ practices in a variety of fields (for example, occupational therapy, dentistry, mental health, pharmacology, gerontology, nursing and so on). The health policy and systems articles usually described or assessed KMb strategies targeting decision-making processes, decision-makers or public health interventions and policies. The continuing education articles assessed training programmes for health professionals aimed at increasing knowledge and skills in a specific field. The articles in the general field described or discussed TMFs and KMb strategies that could be applied to multiple disciplines or contexts. Finally, the articles in the education field described or assessed a KMb strategy targeting education professionals.

Almost half of the articles ( n  = 42, 47.7%) studied KMB strategies implemented in only four countries: Canada, Australia, the United States and the United Kingdom. Countries in South America, the Caribbean, Africa, Asia, the Middle East, China and Europe were underrepresented ( n  = 8, 9.1%). The remaining 34 articles (38.6%) did not specify an implementation context and were mostly theoretical articles. Regarding the authors’ countries of affiliation, Canada, the United States, Australia and the United Kingdom were again the most represented countries, featuring in 85% of the articles ( n  = 75).

What theories, models or frameworks exist in the literature to evaluate KMb strategies?

Several articles proposed a new TMF ( n  = 37, 42.0%), and some articles proposed a logic model specifically developed to evaluate their KMb strategy ( n  = 17, 19.3%). One-third of the articles used existing TMFs ( n  = 28, 31.8%). A few articles only referred to existing TMFs but did not use them to guide a KMb strategy evaluation ( n  = 6, 8.5%).

The identified TMFs were then categorized according to their theoretical approaches (adapted from Nilsen, [ 32 ]) (Table  5 ). Five articles used or proposed more than one TMF, and three TMFs could be classified in two categories. Several articles proposed or used a process model ( n  = 34, 38.6%) or an evaluation framework ( n  = 28, 31.8%); these were the two most frequently identified types of TMFs. Fewer articles proposed or used a logic model ( n  = 17, 19.3%), a determinant framework ( n  = 12, 13.6%) or a classic theory ( n  = 7, 8.0%). The TMFs most often identified in the articles were the RE-AIM framework ( n  = 5, 5.7%), the Knowledge-to-Action framework [ 9 ] ( n  = 4, 4.5%), the Theory of Planned Behavior [ 51 ] ( n  = 3, 3.4%) and the Expanded Outcomes framework for planning and assessing continuing medical education [ 52 ] ( n  = 3, 3.4%). In total, we identified 87 different TMFs in the 88 articles. Only nine TMFS were retrieved in more than one article.

What KMb strategies do the TMFs evaluate (activities, objectives, target audience)?

Thirty-eight articles reported using more than one activity in their KMb strategy. According to the ERIC compilation, “Train and educate stakeholders” activities were the most common, followed by “Develop stakeholder interrelationships” and “Use evaluative and iterative strategies”. Table 6 presents the various types of activities and the number of articles that referred to each.

Of the 88 articles analysed, 18 (20.4%) did not specify a KMb objective. The remaining articles proposed one or more KMb strategy objectives. Specifically, 39 (36.4%) articles had one objective, 15 (17.0%) had two, three (3.4%) had three, and 13 (14.8%) had four or five. Table 7 presents the different types of objectives and the number of times they were identified.

The target audiences for KMb strategies were clearly specified in half of the articles ( n  = 44, 50.0%). Generally, these were empirical articles that targeted specific professionals ( n  = 36, 40.9%) or decision-makers ( n  = 8, 9.1%). Just under one-third of the articles identified a broad target audience (for example, professionals and managers in the health system, a health organization) ( n  = 26, 29.5%). Finally, 18 articles (20.4%) did not specify a target audience for KMb; these were most often theoretical articles.

What are the dimensions and components included in TMFs for evaluating KMb strategies?

The analysis of the identified TMFs revealed many factors of interest relevant for the evaluation of KMb strategies. These specific components were inductively classified into four main dimensions: context, process, effects and impacts (Fig.  2 ). The context dimension refers to the assessment of the conditions in place when the KMb strategy is implemented. These include both the external (that is, sociopolitical, economic, environmental and cultural characteristics) and internal environments (that is, characteristics of organizations, individuals and stakeholder partnerships). These factors are understood to influence the selection and tailoring of a KMb strategy. The process dimension refers to the assessment of the planning, levels and mechanisms of implementation, as well as to the characteristics of the KMb strategy implemented. The effects dimension refers to the assessment of outcomes following the KMb strategy implementation. The potential effects vary depending on the strategy’s objectives and can be either the immediate results of the KMb strategy or short-, medium- and long-term outcomes. The conceptual gradation of effects was generally represented in a similar way in the TMFs analysed, but the temporality of effects could vary. A medium-term outcome in one study could be understood as a long-term outcome in another. However, the majority of authors group these effects into three categories (Gervais et al. 2016: p. 6): (1) short-term effects, measured by success of KMb strategy measured by success of KMb strategy (number of people reached, satisfaction, participation and so on); (2) medium-term effects linked to changes in individual attitude and the use of knowledge; and (3) the long-term effects that result from achieving the KMb objective (for example, improved practices and services, changed collective behaviour, sustainable use of knowledge).

The main evaluation dimensions that emerged from the TMFs analysed

Finally, the impacts dimension refers to the ultimate effects of KMb products or interventions on end users, as measured by the organization (Phipps et al. [ 36 ], p. 34). The evaluation of these ultimate effects can be measured by the integration of a promising practice into organizational routines, by the effects on service users or by the effects on the health and well-being of communities and society in general.

This gradation shows the importance of measuring effects at different points in time, to take account of the time they take to appear and their evolving nature (Gervais et al., 2016: p. 6).

Most of the articles presented the dimensions that should be evaluated, whereas the empirical articles presented the dimensions but also used them in practice to evaluate a KMb strategy. Only five articles (5.7%) did not mention specific dimensions that could be classified.

Table 8 presents both the number of articles that presented dimensions to be evaluated and the number of articles that evaluated them in practice. These results showed that the effects dimension was both the most often named and the most evaluated in practice. The other three dimensions (context, process, impacts), while quite often mentioned as relevant to assess, were less often evaluated in practice. For example, only five articles (5.7%) reported having assessed the impacts dimension.

As previously mentioned, the components relevant for the evaluation of KMb strategies were extracted from the identified TMFs. Table 9 presents these components, which represent the more specific factors of interest for assessing context, process, effects and impacts.

Although often overlooked, the evaluation of KMb strategies is an essential step in guiding organizations seeking to determine whether the expected outcomes of their initiatives are being realized. Evaluation not only allows organizations to make adjustments if the initiatives are not producing the expected results, but also helps them to justify their funding of such initiatives. Evaluation is also essential if the KMb science is to truly inform KMb practice, such that the strategies developed are based on empirical data [ 30 ]. To make KMb evaluation more feasible, evaluation must be promoted and practices improved.

This scoping review meets the first objective of our project, which was to provide an overview of reference frameworks used or proposed for evaluating KM strategies, and to propose a preliminary version of a reference framework for evaluating KM strategies. Several key findings emerged from this scoping review:

Proliferation of theories, models and frameworks, but few frequently used

We are seeing a proliferation of TMFs in KMb and closely related fields [ 132 , 133 ]. Thus, the results of this scoping review support the argument that the conceptual and theoretical development of the field is outpacing its empirical development. Most of the reviewed articles (42.0%) proposed a new TMF rather than using existing ones. Furthermore, we identified relatively few empirical studies (50.0%) that focused on the evaluation of KMb strategies. Consequently, the TMFs used were poorly consolidated, which does not provide a solid empirical foundation to guide the evaluation of KMb strategies. Also, not all the TMFs proposed in the articles were specifically developed for evaluation; some were focused on KMb implementation processes. These may still provide elements to consider for evaluation, although they were not designed to propose specific indicators.

A scoping review published in 2018 identified 596 studies using 159 different KMb TMFs, 95 of which had been used only once [ 11 ]. Many authors reported that these are rarely reused and validated [ 11 , 30 , 33 ] and that it is important to test, refine and integrate existing ones [ 3 , 31 , 34 , 133 ]. A clear, collective and consistent use of existing TMFs is recommended and necessary to advance KMb science and closely related fields [ 12 , 31 ]. The systematic review by Strifler et al. [ 11 ] highlights the diversity of available TMFs and the difficulty users may experience when choosing TMFs to guide their KMb initiatives or evaluation process. Future work should focus on the development of tools to better support users of TMFs, especially those working in organizations. By consolidating a large number of TMFs, the results of this scoping review contribute to these efforts.

The importance of improving evaluation practices for complex multifaceted KMb strategies

Another noteworthy finding was the emphasis on the evaluation of strategies focused on education and professional training for practice improvement (52.3%). Relatively few of the reviewed articles looked at, for example, the evaluation of KMb strategies aimed at informing or influencing decision-making (13.6%), or KMb strategies targeting decision-makers (9.1%). These results reaffirm the importance of conducting more large-scale evaluations of complex and multifaceted KMb strategies. These involve a greater degree of interaction and engagement, are composed of networks of multiple actors, mobilize diverse sources of knowledge and have simultaneous multilevel objectives [ 19 , 134 ].

The fact that some KMb strategies are complex interventions implemented in complex contexts [ 134 ] presents a significant and recurring challenge to their evaluation. Methodological designs, approaches and tools are often ill-suited to capture the short-, medium- and long-term outcomes of KMb strategies, as well as to identify the mechanisms by which these outcomes were produced in a specific context. It is also difficult to link concrete changes in practice and decision-making to tangible longer-term impacts at the population level. Moreover, these impacts can take years to be achieved [ 36 ] and can be influenced by several other factors in addition to KMb efforts [ 2 , 19 , 24 ]. Comprehensive, dynamic and flexible evaluation approaches [ 135 , 136 , 137 ] using mixed methods [ 20 ] appear necessary to understand why, for whom, how, when and in what context KMb strategies achieve their objectives [ 2 , 21 , 25 ]. For instance, realist evaluation, which belongs to theory-based evaluation, may be an approach that addresses issues of causality without sacrificing complexity [ 134 , 138 , 139 ]. This evaluation approach aims to identify the underlying generative mechanisms that can explain how the outcomes were generated and what characteristics of the context affected, or not, those mechanisms. This approach is used to test and refine theory about how interventions with a similar logic of action actually work [ 139 ].

Large heterogeneity of methodologies used in empirical studies

Despite the growth of the KMb field, a recurring issue is the relatively limited number of high-quality studies that evaluate KMb outcomes and impacts. This observation is shared by many of the authors of our scoping articles [ 2 , 22 , 23 , 24 , 25 ]. Only a limited number of empirical articles met the selection criteria ( n  = 44/88) in this scoping review. Synthesizing these studies is challenging due to the diversity of research designs used and the large number of potential evaluation components identified. In addition, most of the identified studies used TMFs and measurement tools that were not validated [ 20 , 29 ] and that were specifically developed for their study [ 16 , 25 , 140 ]. Moreover, these studies did not describe the methods used to justify their choice of evaluation dimensions and components [ 25 ], which greatly hinders the ability to draw inferences and develop generalizable theories through replication in similar studies [ 110 , 140 , 141 , 142 , 143 ]. The lack of a widely used evaluation approach across the field is therefore an important issue [ 16 , 20 ] also highlighted by this scoping review.

Our aim in this review was not to identify specific indicators or measurement tools (for example, questionnaires) for assessing KMb strategies, but rather to describe dimensions and component of TMFs used for KMb evaluation. However, a recent scoping review [ 144 ] looked at measurement tools and revealed that only two general potential tools have been identified to assess KMb activities in any sector or organization: the Level of Knowledge Use Survey (LOKUS) [ 145 ] and the Knowledge Uptake and Utilization Tool (KUUT) [ 95 ]. The authors also assert the importance of developing standardized tools and evaluation processes to facilitate comparison of KMb activities’ outcomes across organizations [ 144 ].

Lack of description and reporting of KMb strategies and evaluation

Another important finding from this review was the sparsity of descriptions of KMb strategies in the published articles. In general, the authors provided little information on the operationalization of their KMb strategies (for example, objectives, target audiences, details of activities implemented, implementation context, expected effects). The KMb strategy objectives and the implemented activities should be carefully selected and empirically, theoretically or pragmatically justified before the evaluation components and specific indicators can be determined [ 146 ].

To improve consistency in the field and to contribute to the development of KMb science, many authors reported the need to better describe and report KMb strategies and their context [ 8 , 54 , 146 , 147 , 148 , 149 , 150 ]. KMb strategies are often inconsistently labelled across studies, poorly described and rarely justified theoretically [ 146 , 150 , 151 ]. It was not possible in this scoping review to associate the evaluation components to be used with the objectives and types of KMb strategies, as too much information was missing in the articles. Over the past 10 years, several guidelines have been proposed to improve the reporting of interventions such as KMb strategies: the “Workgroup for Intervention Development and Evaluation Research (WIDER) recommendations checklist” [ 147 ], the “Standards for Reporting Implementation Studies (StaRI)” [ 150 ] and the “Template for Intervention Description and Replication (TIDieR)” [ 152 ]. These guidelines should be used more often to enhance the reporting of KMb strategies and help advance the field [ 153 ].

Implications for future research

This scoping review provides an overview of potential factors of interest for assessing the context, process, effects and impacts of a KMb strategy. It also proposes a preliminary inventory of potential dimensions and components to consider when planning the evaluation of a KMb strategy. Given the broad spectrum of factors of interest identified across studies, not all of them can be assessed in every context. Rather, they should be targeted according to the objectives of the evaluation, the nature of the KMb strategy and the resources available to conduct the evaluation. Thus, this inventory should not be understood as a prescriptive, normative and exhaustive framework, but rather as a toolbox to identify the most relevant factors to include in the evaluation of a given KMB strategy, and to address a need often expressed by organizations wishing to evaluate their KMb efforts.

Additional work is needed to validate and operationalize these dimensions, to identify relevant measurement tools related to the different components and to see how this inventory could support KMb evaluation practices in organizations.

This scoping review is the first stage of a larger research project aimed at improving organizations’ capacity to evaluate their KMb initiatives by developing an integrative, interdisciplinary and easy-to-use reference framework. In the second phase of the project, the relevance and clarity of the evaluation dimensions identified in the scoping review will be validated through a Delphi study with KMb specialists and researchers. The enriched framework will then be pilot tested in two organizations carrying out and evaluating KMb strategies, to adapt the framework to their needs and to further clarify how the dimensions can be measured in practice. In this third phase, guidance will be provided to help organizations adopt the framework and its support kit. The aim of the project is to go beyond proposing a theoretical framework, and to help build organizations’ capacity to evaluate KT strategies by proposing tools adapted to their realities.

Review limitations

Some limitations of this scoping review should be acknowledged. First, given the numerous different terms used to describe and conceptualize the science of using evidence, it is possible that our search strategy did not capture all relevant publications. However, to limit this risk, we manually searched the reference lists of the selected articles. Second, the literature search was limited to articles published in English or French, and the articles were mostly from high-income countries (for example, North America); therefore, the application of the identified concepts in this scoping review to other contexts should be further explored.

In addition, the search strategy focused on scientific publications to assess progress made in the field of knowledge mobilization strategy evaluation. The grey literature was not examined. It should be considered in future research to complete the overview of evaluation needs in the field of knowledge mobilization.

Finally, the paucity of information in the articles sometimes made it difficult to classify the TMFs according to the taxonomies [ 32 , 44 ], which may have led to possible misinterpretation. However, to limit the risk of errors, the categorization was performed by two reviewers and validated by a third in cases of uncertainty.

Given the increasing demand from organizations for the evaluation of KMb strategies, along with the poorly consolidated KMb research field, a scoping review was needed to identify the range, nature and extent of the literature. This scoping review enabled us to synthesize the breadth of the literature, provide an overview of the many theories, models and frameworks used, and identify and categorize the potential dimensions and components to consider when evaluating KMb initiatives. This scoping review is part of a larger research project, in which the next steps will be to validate the integrative framework and develop a support kit to facilitate its use by organizations involved in KMb.

Availability of data and materials

The dataset supporting the conclusions of this article is included within the article and its additional files.

Abbreviations

• Knowledge mobilization
• Theories, models, and frameworks

Social Sciences and Humanities Research Council. Guidelines for Effective Knowledge Mobilization. 2019. https://www.sshrc-crsh.gc.ca/funding-financement/policies-politiques/knowledge_mobilisation-mobilisation_des_connaissances-eng.aspx Accessed 28 Dec 2022.

Boaz A, Davies H, Fraser A, Nutley S. What works now? evidence-informed policy and practice. Bristol: Policy press; 2019.

Book   Google Scholar  

Curran JA, Grimshaw JM, Hayden JA, Campbell B. Knowledge translation research: the science of moving research into policy and practice. J Contin Educ Heal Prof. 2011;31(3):174–80.

Article   Google Scholar  

Global Commission on Evidence. The Evidence Commission report: A wake-up call and path forward for decision-makers, evidence intermediaries, and impact-oriented evidence producers. McMaster University; 2022 p. 144. https://www.mcmasterforum.org/networks/evidence-commission/report/english

Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510–20.

Article   PubMed   PubMed Central   Google Scholar  

Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS ONE. 2011;6(7): e21704.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Straus SE, Tetroe J, Graham ID, editors. Knowledge translation in health care: moving from evidence to practice. 2nd ed. Chichester, West Sussex ; Hoboken, NJ: Wiley/BMJ Books; 2013, 406

Barwick M, Dubrowski R, Petricca K. Knowledge translation: The rise of implementation. 2020; https://ktdrr.org/products/kt-implementation/KT-Implementation-508.pdf

Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Continuing Educ Health Professions. 2006;26(1):13–24.

Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7(1):50.

Strifler L, Cardoso R, McGowan J, Cogo E, Nincic V, Khan PA, et al. Scoping review identifies significant number of knowledge translation theories, models, and frameworks with limited use. J Clin Epidemiol. 2018;100:92–102.

Article   PubMed   Google Scholar  

Esmail R, Hanson HM, Holroyd-Leduc J, Brown S, Strifler L, Straus SE, et al. A scoping review of full-spectrum knowledge translation theories, models, and frameworks. Implement Sci. 2020;15(1):11.

McKibbon KA, Lokker C, Wilczynski NL, Ciliska D, Dobbins M, Davis DA, et al. A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel? Implement Sci. 2010;5(1):16.

Fonds de recherche du Québec. Stratégie de mobilisation des connaissances 2014–2017. 2014. https://frq.gouv.qc.ca/en/mobilization-of-knowledge/ . Accessed 28 Dec 2022.

Gervais MJ, Souffez K, Ziam S. Quel impact avons-nous ? Vers l’élaboration d’un cadre pour rendre visibles les retombées du transfert des connaissances. TUC Revue francophone de recherche sur le transfert et l’utilisation des connaissances. 2016;1(2):21.

Google Scholar  

Williams NJ, Beidas RS. Annual research review: the state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field. J Child Psychol Psychiatr. 2019;60(4):430–50.

Mitton C, Adair CE, Mckenzie E, Patten SB, Perry BW. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007;85(4):729–68.

Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14(1):2.

Fazey I, Bunse L, Msika J, Pinke M, Preedy K, Evely AC, et al. Evaluating knowledge exchange in interdisciplinary and multi-stakeholder research. Glob Environ Chang. 2014;25:204–20.

Gervais MJ, Marion C, Dagenais C, Chiocchio F, Houlfort N. Dealing with the complexity of evaluating knowledge transfer strategies: guiding principles for developing valid instruments. Res Eval. 2016;25(1):62–9.

Reed MS, Bryce R, Machen R. Pathways to policy impact: a new approach for planning and evidencing research impact. Evid policy. 2018;14(3):431–58.

Kim C, Wilcher R, Petruney T, Krueger K, Wynne L, Zan T. A research utilisation framework for informing global health and development policies and programmes. Health Res Policy Sys. 2018;16(1):9.

Langer L, Tripney J, Gough D University of London, Social Science Research Unit, Evidence for Policy and Practice Information and Co-ordinating Centre. The science of using science: researching the use of research evidence in decision-making. 2016.

Rajić A, Young I, McEwen SA. Improving the utilization of research knowledge in agri-food public health: a mixed-method review of knowledge translation and transfer. Foodborne Pathog Dis. 2013;10(5):397–412.

Scarlett J, Forsberg BC, Biermann O, Kuchenmüller T, El-Khatib Z. Indicators to evaluate organisational knowledge brokers: a scoping review. Health Res Policy Syst. 2020;18(1):93.

Bornbaum CC, Kornas K, Peirson L, Rosella LC. Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge translation in health-related settings: a systematic review and thematic analysis. Implement Sci. 2015;10(1):162.

Sarkies MN, Bowles KA, Skinner EH, Haas R, Lane H, Haines TP. The effectiveness of research implementation strategies for promoting evidence-informed policy and management decisions in healthcare: a systematic review. Implement Sci. 2017;12(1):132.

Scott SD, Albrecht L, O’Leary K, Ball GD, Hartling L, Hofmeyer A, et al. Systematic review of knowledge translation strategies in the allied health professions. Implement Sci. 2012;7(1):70.

Dagenais C, Malo M, Robert É, Ouimet M, Berthelette D, Ridde V. Knowledge transfer on complex social interventions in public health: a scoping study. PLoS ONE. 2013;8(12): e80233.

Davies HT, Powell AE, Nutley SM. Mobilising knowledge to improve UK health care: learning from other countries and other sectors – a multimethod mapping study. Health Serv Deliv Res. 2015;3(27):1–190.

Damschroder LJ. Clarity out of chaos: use of theory in implementation research. Psychiatry Res. 2020;283: 112461.

Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):53.

Ellen ME, Panisset U, Araujo de Carvalho I, Goodwin J, Beard J. A knowledge translation framework on ageing and health. Health Policy. 2017;121(3):282–91.

Wensing M, Bosch M, Grol R. Developing and selecting interventions for translating knowledge to action. CMAJ. 2010;182(2):E85–8.

Bennet A, Bennet D, Fafard K, Fonda M, Lomond T, Messier L, et al. Knowledge mobilization in the social sciences and humanities: moving from research to action. Frost: MQI Press; 2007.

Phipps D, Cummins J, Pepler D, Craig W, Cardinal S. The Co-produced Pathway to Impact Describes Knowledge Mobilization Processes. JCES. 2016;9(1). https://jces.ua.edu/articles/258 . Accessed 17 Nov 2022.

Birken SA, Rohweder CL, Powell BJ, Shea CM, Scott J, Leeman J, et al. T-CaST: an implementation theory comparison and selection tool. Implement Sci. 2018;13(1):143.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Moullin JC, Sabater-Hernandez D, Fernandez-Llimos F, Benrimoj SI. A systematic review of implementation frameworks of innovations in healthcare and resulting generic implementation framework. Health Res Policy Syst. 2015;13(101170481):16.

Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synthesis Methods. 2014;5(4):371–85.

Farkas M, Jette AM, Tennstedt S, Haley SM, Quinn V. Knowledge dissemination and utilization in gerontology: an organizing framework. Gerontologist. 2003;43:47–56.

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):21.

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10(1):109.

Smith JD, Li DH, Rafferty MR. The implementation research logic model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci. 2020;15(1):84.

Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.

Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7:149–58.

Sketris IS, Carter N, Traynor RL, Watts D, Kelly K, following contributing members of the CNODES Knowledge Translation Team: Pierre Ernst JG Brenda Hemmelgarn, Colleen Metge, Michael Paterson, Robert Platt W and Gary Teare. Building a framework for the evaluation of knowledge translation for the Canadian Network for Observational Drug Effect Studies. Pharmacoepidemiol Drug Saf. 2020;29 Suppl 1(d0r, 9208369):8–25.

Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006;27(2):237–46.

Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50(2):179–211.

Moore DE, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1–15.

Tschida JE, Drahota A. Fidelity to the ACT SMART Toolkit: an instrumental case study of implementation strategy fidelity. Implement Sci Commun. 2023;4(1):52.

Colquhoun H, Leeman J, Michie S, Lokker C, Bragge P, Hempel S, et al. Towards a common terminology: a simplified framework of interventions to promote and integrate evidence into health practices, systems, and policies. Implement Sci. 2014;9(1):781.

Bertone MP, Meessen B, Clarysse G, Hercot D, Kelley A, Kafando Y, et al. Assessing communities of practice in health policy: a conceptual framework as a first step towards empirical research. Health Res Policy Sys. 2013;11(1):39.

Gagliardi AR, Legare F, Brouwers MC, Webster F, Wiljer D, Badley E, et al. Protocol: developing a conceptual framework of patient mediated knowledge translation, systematic review using a realist approach. Implement Sci. 2011;6(101258411):25.

Sargeant J, Borduas F, Sales A, Klein D, Lynn B, Stenerson H. CPD and KT: models used and opportunities for synergy. J Contin Educ Heal Prof. 2011;31(3):167–73.

Stetler CB, Ritchie J, Rycroft-Malone J, Schultz A, Charns M. Improving quality of care through routine, successful implementation of evidence-based practice at the bedside: an organizational case study protocol using the Pettigrew and Whipp model of strategic change. Implement Sci. 2007;2(101258411):3.

Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst. 2012;10(101170481):21.

Dadich A. From bench to bedside: methods that help clinicians use evidence-based practice. Aust Psychol. 2010;45(3):197–211.

Brown P, Bahri P. Engagement’ of patients and healthcare professionals in regulatory pharmacovigilance: establishing a conceptual and methodological framework. Eur J Clin Pharmacol. 2019;75(9):1181–92.

Article   CAS   PubMed   Google Scholar  

Dobbins M, Ciliska D, Cockerill R, Barnsley J, DiCenso A. A framework for the dissemination and utilization of research for health-care policy and practice. Worldviews Evid Based Nurs Presents Arch Online J Knowl Synthesis Nurs. 2002;9(1):149–60.

Gagliardi AR, Brouwers MC, Bhattacharyya OK. The guideline implementability research and application network (GIRAnet): an international collaborative to support knowledge exchange: study protocol. Implement Sci. 2012;7(101258411):26.

Brooks SP, Zimmermann GL, Lang M, Scott SD, Thomson D, Wilkes G, et al. A framework to guide storytelling as a knowledge translation intervention for health-promoting behaviour change. Implement sci commun. 2022;3(1):35.

Cullen L, Hanrahan K, Edmonds SW, Reisinger HS, Wagner M. Iowa implementation for sustainability framework. Implement Sci. 2022;17(1):1.

Labbé D, Mahmood A, Miller WC, Mortenson WB. Examining the impact of knowledge mobilization strategies to inform urban stakeholders on accessibility: a mixed-methods study. Int J Environ Res Public Health. 2020;17(5):1561.

Straus SE, Tetroe J, Graham ID, Zwarenstein M, Bhattacharyya O, Shepperd S. Monitoring use of knowledge and evaluating outcomes. Can Med Assoc J. 2010;182(2):E94–8.

Bennett S, Whitehead M, Eames S, Fleming J, Low S, Caldwell E. Building capacity for knowledge translation in occupational therapy: learning through participatory action research. BMC Med Educ. 2016;16(1):257.

Brown C, Rogers S. Measuring the effectiveness of knowledge creation as a means of facilitating evidence-informed practice in early years settings in one London Borough. Lond Rev Educ. 2014;12(3):245–60.

Talbott E, De Los RA, Kearns DM, Mancilla-Martinez J, Wang M. Evidence-based assessment in special education research: advancing the use of evidence in assessment tools and empirical processes. Except Child. 2023;89(4):467–87.

Rosella LC, Bornbaum C, Kornas K, Lebenbaum M, Peirson L, Fransoo R, et al. Evaluating the process and outcomes of a knowledge translation approach to supporting use of the Diabetes Population Risk Tool (DPoRT) in public health practice. Canadian J Program Eval. 2018;33(1):21–48.

Couineau AL, Forbes D. Using predictive models of behavior change to promote evidence-based treatment for PTSD. Psychol Trauma Theory Res Pract Policy. 2011;3(3):266–75.

Dufault M. Testing a collaborative research utilization model to translate best practices in pain management. Worldviews Evid Based Nurs. 2004;1:S26-32.

Beckett K, Farr M, Kothari A, Wye L, le May A. Embracing complexity and uncertainty to create impact: exploring the processes and transformative potential of co-produced research through development of a social impact model. Health Res Policy Syst. 2018;16(1):118.

Kramer DM, Wells RP, Carlan N, Aversa T, Bigelow PP, Dixon SM, et al. Did you have an impact? A theory-based method for planning and evaluating knowledge-transfer and exchange activities in occupational health and safety. Int J Occup Saf Ergon. 2013;19(1):41–62.

Duhamel F, Dupuis F, Turcotte A, Martinez AM, Goudreau J. Integrating the illness beliefs model in clinical practice: a family systems nursing knowledge utilization model. J FAM NURS. 2015;21(2):322–48.

Wimpenny P, Johnson N, Walter I, Wilkinson JE. Tracing and identifying the impact of evidence-use of a modified pipeline model. Worldviews Evid Based Nurs. 2008;5(1):3–12.

Ward V, Smith S, House A, Hamer S. Exploring knowledge exchange: a useful framework for practice and policy. Soc Sci Med. 2012;74(3):297–304.

Grooten L, Vrijhoef HJM, Alhambra-Borras T, Whitehouse D, Devroey D. The transfer of knowledge on integrated care among five European regions: a qualitative multi-method study. BMC Health Serv Res. 2020;20(1):11.

Stetler CB. Updating the Stetler Model of research utilization to facilitate evidence-based practice. Nurs Outlook. 2001;49(6):272–9.

Ward V. Why, whose, what and how? A framework for knowledge mobilisers. Evid Policy J Res Debate Pract. 2017;13(3):477–97.

Levin RF, Fineout-Overholt E, Melnyk BM, Barnes M, Vetter MJ. Fostering evidence-based practice to improve nurse and cost outcomes in a community health setting: a pilot test of the advancing research and clinical practice through close collaboration model. Nurs Adm Q. 2011;35(1):21–33.

Currie M, King G, Rosenbaum P, Law M, Kertoy M, Specht J. A model of impacts of research partnerships in health and social services. Eval Program Plann. 2005;28(4):400–12.

Richard L, Chiocchio F, Essiembre H, Tremblay MC, Lamy G, Champagne F, et al. Communities of practice as a professional and organizational development strategy in local public health organizations in Quebec, Canada: an evaluation model. Healthc Policy. 2014;9(3):26–39.

PubMed   PubMed Central   Google Scholar  

Rycroft-Malone J, Wilkinson J, Burton CR, Harvey G, McCormack B, Graham I, et al. Collaborative action around implementation in collaborations for leadership in applied health research and care: towards a programme theory. J Health Serv Res Policy. 2013;18(3 Suppl):13–26.

Gagliardi AR, Fraser N, Wright FC, Lemieux-Charles L, Davis D. Fostering knowledge exchange between researchers and decision-makers: exploring the effectiveness of a mixed-methods approach. Health Policy. 2008;86(1):53–63.

Paquette-Warren J, Harris SB, Naqshbandi Hayward M, Tompkins JW. Case study of evaluations that go beyond clinical outcomes to assess quality improvement diabetes programmes using the Diabetes Evaluation Framework for Innovative National Evaluations (DEFINE). J Eval Clin Pract. 2016;22(5):644–52.

Paquette-Warren J, Tyler M, Fournie M, Harris SB. The diabetes evaluation framework for innovative national evaluations (DEFINE): construct and content validation using a modified Delphi method. Can J diabetes. 2017;41(3):281–96.

Abbot ML, Lee KK, Rossiter MJ. Evaluating the effectiveness and functionality of professional learning communities in adult ESL Programs. TESL Canada J. 2018;35(2):1–25.

Ho K, Bloch R, Gondocz T, Laprise R, Perrier L, Ryan D, et al. Technology-enabled knowledge translation: frameworks to promote research and practice. J Contin Educ Heal Prof. 2004;24(2):90–9.

Yu X, Hu D, Li N, Xiao Y. Comprehensive evaluation on teachers’ knowledge sharing behavior based on the improved TOPSIS method. Comput Intell Neurosci. 2022;2022(101279357):2563210.

Arora S, Kalishman SG, Thornton KA, Komaromy MS, Katzman JG, Struminger BB, et al. Project ECHO: a telementoring network model for continuing professional development. J Contin Educ Health Prof. 2017;37(4):239–44.

Smidt A, Balandin S, Sigafoos J, Reed VA. The Kirkpatrick model: a useful tool for evaluating training outcomes. J Intellect Dev Disabil. 2009;34(3):266–74.

Jeffs L, Sidani S, Rose D, Espin S, Smith O, Martin K, et al. Using theory and evidence to drive measurement of patient, nurse and organizational outcomes of professional nursing practice. Int J Nurs Pract. 2013;19(2):141–8.

Skinner K. Developing a tool to measure knowledge exchange outcomes. Can J Program Eval. 2007;22(1):49–75.

Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Serv Res Policy. 2003;8(3):165–70.

Boyko JA, Lavis JN, Abelson J, Dobbins M, Carter N. Deliberative dialogues as a mechanism for knowledge translation and exchange in health systems decision-making. Soc Sci Med. 2012;75(11):1938–45.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.

Gainforth HL, Latimer-Cheung AE, Athanasopoulos P, Martin Ginis KA. Examining the feasibility and effectiveness of a community-based organization implementing an event-based knowledge mobilization initiative to promote physical activity guidelines for people with spinal cord injury among support personnel. Health Promot Pract. 2015;16(1):55–62.

Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019. https://doi.org/10.3389/fpubh.2019.00064 .

Shelton RC, Chambers DA, Glasgow RE. An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Front Public Health. 2020;8(101616579):134.

Bender BG, Simmons B, Konkoly N, Liu AH. The asthma toolkit bootcamp to improve rural primary care for pediatric asthma. J Allergy Clin Immunol Pract. 2021;9(8):3091-3097.e1.

de la Garza Iga FJ, Mejia Alvarez M, Cockroft JD, Rabin J, Cordon A, Elias Rodas DM, et al. Using the project ECHO TM model to teach mental health topics in rural Guatemala: an implementation science-guided evaluation. Int J Soc Psychiatry. 2023;69(8):2031–41.

Alkin M, Taut S. Unbundling evaluation use. Stud Educ Eval. 2003;29(1):1–12.

Varallyay NI, Langlois EV, Tran N, Elias V, Reveiz L. Health system decision-makers at the helm of implementation research: development of a framework to evaluate the processes and effectiveness of embedded approaches. Health Res Policy Syst. 2020;18(1):64.

McCabe KE, Wallace A, Crosland A. A model for collaborative working to facilitate knowledge mobilisation in public health. Evid Policy. 2015;11(4):559–76.

Gonzales R, Handley MA, Ackerman S, O’sullivan PS. A framework for training health professionals in implementation and dissemination science. Acad Med. 2012;87(3):271–8.

Edgar L, Herbert R, Lambert S, MacDonald JA, Dubois S, Latimer M. The joint venture model of knowledge utilization: a guide for change in nursing. Nurs Leadersh. 2006;9(2):41–55.

Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ. A Guide for applying a revised version of the PARIHS framework for implementation. Implement Sci. 2011;6(101258411):99.

Brennan SE, Cumpston M, Misso ML, McDonald S, Murphy MJ, Green SE. Design and formative evaluation of the policy liaison initiative: a long-term knowledge translation strategy to encourage and support the use of cochrane systematic reviews for informing. Evid Policy. 2016;12(1):25–52.

Hinchcliff R, Senserrick T, Travaglia J, Greenfield D, Ivers R. The enhanced knowledge translation and exchange framework for road safety: a brief report on its development and potential impacts. Inj Prev. 2017;23(2):114–7.

Ye J, Woods D, Bannon J, Bilaver L, Kricke G, McHugh M, et al. Identifying contextual factors and strategies for practice facilitation in primary care quality improvement using an informatics-driven model: framework development and mixed methods case study. JMIR Hum Factors. 2022;9(2): e32174.

Brangan J, Quinn S, Spirtos M. Impact of an evidence-based practice course on occupational therapist’s confidence levels and goals. Occup Ther Health Care. 2015;29(1):27–38.

Bonetti D, Johnston M, Pitts NB, Deery C, Ricketts I, Tilley C, et al. Knowledge may not be the best target for strategies to influence evidence-based practice: using psychological models to understand RCT effects. Int J Behav Med. 2009;16(3):287–93.

Buckley LL, Goering P, Parikh SV, Butterill D, Foo EKH. Applying a “stages of change” model to enhance a traditional evaluation of a research transfer course. J Eval Clin Pract. 2003;9(4):385–90.

Boyko JA, Lavis JN, Dobbins M, Souza NM. Reliability of a tool for measuring theory of planned behaviour constructs for use in evaluating research use in policymaking. Health Res Policy Syst. 2011;24(9):29.

Imani-Nasab MH, Yazdizadeh B, Salehi M, Seyedin H, Majdzadeh R. Validity and reliability of the Evidence Utilisation in Policymaking Measurement Tool (EUPMT). Health Res Policy Syst. 2017;15(1):66.

Dwan KM, McInnes P, Mazumdar S. Measuring the success of facilitated engagement between knowledge producers and users: a validated scale. Evid Policy. 2015;11(2):239–52.

Haynes A, Rowbotham S, Grunseit A, Bohn-Goldbaum E, Slaytor E, Wilson A, et al. Knowledge mobilisation in practice: an evaluation of the Australian Prevention Partnership Centre. Health Res Policy Sys. 2020;18(1):13.

Haines M, Brown B, Craig J, D’Este C, Elliott E, Klineberg E, et al. Determinants of successful clinical networks: the conceptual framework and study protocol. Implement Sci. 2012;7(101258411):16.

Ko LK, Jang SH, Friedman DB, Glanz K, Leeman J, Hannon PA, et al. An application of the science impact framework to the cancer prevention and control research network from 2014–2018. Prev Med. 2019;12: 105821.

Leeman J, Sommers J, Vu M, Jernigan J, Payne G, Thompson D, et al. An evaluation framework for obesity prevention policy interventions. Prev Chronic Dis. 2012;9(101205018):E120.

Pettman TL, Armstrong R, Waters E, Allender S, Love P, Gill T, et al. Evaluation of a knowledge translation and exchange platform to advance non-communicable disease prevention. Evid Policy. 2016;12(1):109–26.

Yearwood AC. Applying a logical theory of change for strengthening research uptake in policy: a case study of the Evidence Informed Decision Making Network of the Caribbean. Rev Panam Salud Publica. 2018;42: e91.

Thomson D, Brooks S, Nuspl M, Hartling L. Programme theory development and formative evaluation of a provincial knowledge translation unit. Health Res Policy Syst. 2019;17(1):40.

Garad R, Kozica-Olenski S, Teede HJ. Evaluation of a center of research excellence in polycystic ovary syndrome as a large-scale collaborative research translation initiative, including evaluating translation of guideline impact. Semin Reprod Med. 2018;36(1):42–9.

Reddy S, Wakerman J, Westhorp G, Herring S. Evaluating impact of clinical guidelines using a realist evaluation framework. J Eval Clin Pract. 2015;21(6):1114–20.

Van Eerd D, Moser C, Saunders R. A research impact model for work and health. Am J Ind Med. 2021;64(1):3–12.

Yip O, Huber E, Stenz S, Zullig LL, Zeller A, De Geest SM, et al. A contextual analysis and logic model for integrated care for frail older adults living at home: The INSPIRE Project. Int J Integr Care. 2021;21(2):9.

Guo R, Bain BA, Willer J. Application of a logic model to an evidence-based practice training program for speech-language pathologists and audiologists. J Allied Health. 2011;40(1):e23–8.

PubMed   Google Scholar  

McDonald S, Turner T, Chamberlain C, Lumbiganon P, Thinkhamrop J, Festin MR, et al. Building capacity for evidence generation, synthesis and implementation to improve the care of mothers and babies in South East Asia: methods and design of the SEA-ORCHID Project using a logical framework approach. BMC Med Res Methodol. 2010;10(100968545):61.

Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50.

Wensing M, Grol R. Knowledge translation in health: how implementation science could contribute more. BMC Med. 2019;17(1):88.

Kreindler SA. Advancing the evaluation of integrated knowledge translation. Health Res Policy Sys. 2018;16(1):104.

Best A, Holmes B. Systems thinking, knowledge and action: towards better models and methods. Evid Policy. 2010;6(2):145–59.

van Mil HGJ, Foegeding EA, Windhab EJ, Perrot N, van der Linden E. A complex system approach to address world challenges in food and agriculture. Trends Food Sci Technol. 2014;40(1):20–32.

Wehrens R. Beyond two communities – from research utilization and knowledge translation to co-production? Public Health. 2014;128(6):545–51.

Ridde V, Pérez D, Robert E. Using implementation science theories and frameworks in global health. BMJ Glob Health. 2020;5(4): e002269.

Salter KL, Kothari A. Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci. 2014;9(1):115.

Van Eerd D, Cole D, Keown K, Irvin E, Kramer D, Gibson B, et al. Report on knowledge transfer and exchange practices: A systematic review of the quality and types of instruments used to assess KTE implementation and impact. Toronto: Institute for Work & Health; 2011 p. 130. https://www.iwh.on.ca/sites/iwh/files/iwh/reports/iwh_sys_review_kte_evaluation_tools_2011_rev.pdf

Dobbins M, Robeson P, Ciliska D, Hanna S, Cameron R, O’Mara L, et al. A description of a knowledge broker role implemented as part of a randomized controlled trial evaluating three knowledge translation strategies. Implement Sci. 2009;4(1):23.

Rychetnik L, Bauman A, Laws R, King L, Rissel C, Nutbeam D, et al. Translating research for evidence-based public health: key concepts and future directions. J Epidemiol Community Health. 2012;66(12):1187–92.

Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108.

Bhawra J, Skinner K. Examination of tools associated with the evaluation of knowledge uptake and utilization: a scoping review. Eval Program Plann. 2020;83: 101875.

Lane JP, Stone VI, Nobrega A, Tomita M. Level Of Knowledge Use Survey (LOKUS): a validated instrument for tracking knowledge uptake and use. Stud Health Technol Inform. 2015;217:106–10.

Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139.

Albrecht L, Archibald M, Arseneau D, Scott SD. Development of a checklist to assess the quality of reporting of knowledge translation interventions using the Workgroup for Intervention Development and Evaluation Research (WIDER) recommendations. Implement Sci. 2013;8(1):52.

Bragge P, Grimshaw JM, Lokker C, Colquhoun H, Albrecht L, Baron J, et al. AIMD - a validated, simplified framework of interventions to promote and integrate evidence into health practices, systems, and policies. BMC Med Res Methodol. 2017;17(1):38.

Kastner M, Makarski J, Hayden L, Lai Y, Chan J, Treister V, et al. Improving KT tools and products: development and evaluation of a framework for creating optimized, Knowledge-activated Tools (KaT). Implement Sci Commun. 2020;1(1):47.

Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;6(356): i6795.

Lokker C, McKibbon KA, Colquhoun H, Hempel S. A scoping review of classification schemes of interventions to promote and integrate evidence into practice in healthcare. Implement Sci. 2015;10(1):27.

Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;7(348): g1687.

Wilson PM, Sales A, Wensing M, Aarons GA, Flottorp S, Glidewell L, et al. Enhancing the reporting of implementation research. Implement Sci. 2017;12(1):13.

Download references

Acknowledgements

We wish to thank Julie Desnoyers for designing and implementing the search strategy, Gabrielle Legendre for her contribution in the screening phase and Karine Souffez and Caroline Tessier for their input during the project.

This project was supported by an Insight Grant from the Social Sciences and Humanities Research Council of Canada (SSHRC) and by the Équipe RENARD (FRQ-SC). The funding bodies had no role in the conduct of this scoping review.

Author information

Authors and affiliations.

School of Business Administration, Université TÉLUQ, Montreal, Canada

Saliha Ziam & Laura Justine Chouinard

Department of School and Social Adaptation Studies, Faculty of Education, Université de Sherbrooke, Sherbrooke, Canada

Sèverine Lanoue, Esther McSween-Cadieux, Julie Lane & Ollivier Prigent

Department of Psychology, Université du Québec à Montréal, Montreal, Canada

Mathieu-Joël Gervais

Centre RBC d’expertise Universitaire en Santé Mentale, Université de Sherbrooke, Sherbrooke, Canada

School of Rehabilitation, Faculty of Medicine, Université de Montréal, Montreal, Canada

Dina Gaid & Quan Nha Hong

Department of Psychology, Université de Montréal, Montreal, Canada

Christian Dagenais

Université Paris Cité, IRD (Institute for Research on Sustainable Development, CEPED, Paris, France

Valéry Ridde

Institute of Health and Development (ISED), Cheikh Anta Diop University, Dakar, Senegal

Public Health Intelligence and Knowledge Translation Division, Public Health Agency of Canada, Ottawa, Canada

Emmanuelle Jean

Coordinator of the Interregional Consortium of Knowledge in Health and Social Services (InterS4), Rimouski, Canada

France Charles Fleury

You can also search for this author in PubMed   Google Scholar

Contributions

SZ, MJG, EMC, JL, CD, EJ, KS, VR and CT were involved in developing and designing the scoping review. EMC, MJG and GL (collaborator) screened articles in duplicate. SL, DG, LJC and OP extracted data from the included articles. SL and DG synthesized the data. SL, SZ and EMC drafted the manuscript. SZ led the project, supervised and assisted the research team at every stage, and secured the funding. All authors provided substantive feedback and approved the manuscript prior to submission.

Corresponding author

Correspondence to Saliha Ziam .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Keywords and search strategy.

Additional file 2.

Summary of included articles.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ziam, S., Lanoue, S., McSween-Cadieux, E. et al. A scoping review of theories, models and frameworks used or proposed to evaluate knowledge mobilization strategies. Health Res Policy Sys 22 , 8 (2024). https://doi.org/10.1186/s12961-023-01090-7

Download citation

Received : 16 June 2023

Accepted : 07 December 2023

Published : 10 January 2024

DOI : https://doi.org/10.1186/s12961-023-01090-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Knowledge translation
• Scoping review

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

  African Journal of Empirical Research Journal / African Journal of Empirical Research / About the Journal (function() { function async_load(){ var s = document.createElement('script'); s.type = 'text/javascript'; s.async = true; var theUrl = 'https://www.journalquality.info/journalquality/ratings/2404-www-ajol-info-ajempr'; s.src = theUrl + ( theUrl.indexOf("?") >= 0 ? "&" : "?") + 'ref=' + encodeURIComponent(window.location.href); var embedder = document.getElementById('jpps-embedder-ajol-ajempr'); embedder.parentNode.insertBefore(s, embedder); } if (window.attachEvent) window.attachEvent('onload', async_load); else window.addEventListener('load', async_load, false); })();  

The  African Journal of Empirical Research (AJERNET)  is hosted by  AJER Publishing ,  a network of academic and research publishers headquartered in  Nairobi, Kenya . AJER Publishing has a strong presence in the academic community, with a wide network of reviewers, editors, and contributors from esteemed universities, experienced researchers, and reputable research institutions globally.

Mission Statement

AJERNET aims to provide a dynamic platform for researchers, students, and professors with the view of facilitating access to the latest multidisciplinary literature, encouraging research reviews, and promoting the sharing of expertise among academics worldwide.

Scope and Operational Approach

Our focus is on multidisciplinary empirical research, meta-analysis, and systematic reviews from social, pure and applied science disciplines. AJERNET maintains high standards through a meticulous peer-review process to ensure that only top-quality research is published. We believe in open access to knowledge and all content is published on an open-access basis, allowing for unfettered access, distribution, and reproduction of published works, provided proper citation is given.

Global Reach

We distribute all published content to thousands of libraries worldwide, ensuring that research gains international exposure. We provide comprehensive support and guidance to authors wherever they may be throughout the publishing process. In cases where submitted work doesn't initially meet our publication standards, we offer assistance and time for revisions.

Current Issue: Vol. 4 No. 2 (2023)

Published: 2023-09-08

Students’ Perception of Teacher Classroom Effectiveness and Academic Achievement: A Case of Secondary School Students in Hamisi Sub-County, Kenya

Effect of leadership independence on perceived levels of corruption in the procurement department in the judiciary of kenya, socio-cultural dynamics influencing domestic violence in nyeri county, kenya, effectiveness of guidance and counselling on depression among adolescents in schools in kakamega county, kenya, nature of reintegration programmes for returnee terrorist fighters in mombasa county, kenya, nature of drug and substance abuse in kilifi and mombasa counties of coastal kenya, health managers’ utilization of evidence to inform annual health sector planning and budgeting process in bungoma county, kenya, effect of cashless reform on revenue collection performance in kakamega county, effects of mau forest evictions on human security of the ogiek community, kenya, challenges and opportunities resulting from mau forest evictions of ogiek community, kenya, a critical review on working capital management and financial distress in chartered public universities in kenya, knowledge and attitude of health managers towards the annual health sector planning and budgeting process in bungoma county, kenya, effectiveness of non-custodial reintegration programmes on welfare of returnee terrorist fighters in mombasa, kenya, selected human resource management practices and employee retention: a case of deposit taking microfinance institutions in nairobi, kenya, effect of corporate culture on the performance of the kenya commercial bank, influence of statist inter-ethnic political leadership transformation in the management of inter-ethnic conflicts in bungoma county, kenya, school administration support systems for educational technology adoption and students’ academic achievement in secondary schools in kenya, resource availability and its role in promoting participation in non-formal curricular activities in kenyan secondary schools, impact of universal old age benefit to revenue and poverty reduction in tanzania, effect of credit monitoring practices on loan nonperformance among microfinance institutions in nairobi county, kenya, examining youth participation in local governance: the case of selected districts in ghana, effect of integrated financial management information reporting system on financial management in county governments, kenya, effect of integrated financial management information revenue systems on financial management in county governments, kenya, post covid-19 analysis of the status of online infrastructure use by public universities in kenya: the case of masinde muliro university of science and technology, effect of quality of financial reporting on performance of manufacturing firms listed in the nairobi securities exchange in kenya, effect of rehabilitation on drawings by learners in shikusa borstal institution, kakamega county, influence of liquidity risk management on financial performance of commercial banks in kenya, christian religious studies teachers’ content knowledge: do areas of specialisation, teaching experience, professional qualification and academic qualification matter, evaluation of government programs in mitigating drug and substance abuse in kilifi and mombasa counties, kenya, effect of government capital expenditure on tourism sector growth in kenya, market innovation and competitiveness of commercial banks in kenya, exploring eastern africa's landscape: a scoping study on an independent civil society-un counter-terrorism engagement mechanism, challenges and opportunities of domestic violence in nyeri county, kenya, influence of audit committee meetings on the quality of audit reports of saccos in homabay county, kenya, how technical and governance factors influence financial sustainability of community-managed borehole water kiosk service in kisumu county, republic of kenya, effects of exchange rate on performance of equity funds in kenya, regional factors influencing bilateral trade between kenya and south africa, evidence of climate change and seasonal agricultural drought in kakamega south sub-county, kakamega county, kenya, influence of school physical facilities on academic self-concept in informal settlements in kenya, teacher-learner interactions and academic self-concept in informal settlement primary schools in kenya, influence of governance structure on the financial performance of microfinance banks in kenya, impact of asset quality compliance on financial performance of nairobi securities exchange listed commercial banks, urban proximity and its influence on peri urban land use pattern: a case of kimilili town, bungoma county, kenya, effect of pre-contracting procurement practices on supply chain performance of road authorities in kakamega county, kenya, continuance commitment and employee performance of non-academic staff in public universities in western region of kenya, effect of instructional supervision on utilization of teaching and learning resources for implementation of cbc in pre-primary classrooms in webuye east sub-county, bungoma county, kenya, how household water use characteristics and sense of water kiosk ownership influence financial sustainability of community-managed borehole water kiosk service in kisumu county, kenya, influence of debt restructuring strategy implementation on performance of selected commercial banks in kisumu kenya, incentivizing employee pro-environmental behaviour: harnessing the potential of green rewards, effect of provision of non-audit services on internal audit effectiveness in public technical and vocational education and training (tvet) institutions in western region, kenya, determinants of negotiated wages in the kenyan banking industry, effects of innovation culture on the performance of genge music production companies in starehe sub-county, nairobi county, kenya, influence of internal audit risk planning practices on the financial performance of saccos in the western region of kenya, school budget development and financial accountability of public secondary schools in khwisero sub county, kenya, causes and consequences of post-election violence in kenya, effects of teachers’ distribution and transfer on student’s academic performance in public day secondary schools in naivasha sub-county, kenya, analysing the impact of sudden capital inflow shocks on kenya's economic growth: a quarterly time series analysis (2008 - 2022), exploring teachers’ views in using code-switching for classroom interaction in secondary schools in temeke district, tanzania, role of chinese economic diplomacy in fostering infrastructural development in kenya since 1963, the response of the seventh-day adventist church on management of youth drop-out rate in nyamira county, kenya, comparison of the performance of logistic regression model in the presence and absence of mediation, impact of women empowerment on career advancement of female engineers, trends in selection of agriculture subject among students in secondary schools in bungoma and kakamega counties, kenya, analyzing the impact of institutional infrastructure on learner participation in distance learning: a kenyan perspective, impact of funding on performance of agricultural extension services in bomet, kenya: case of community-based organizations, the impact of resource utilization practices on inter-ethnic conflicts in nakuru county, kenya, effect of expansion decisions on the financial performance of saccos in bungoma county, kenya, supply chain information systems and service delivery of the public health care sector in the county governments of western region, kenya, effect of leadership transparency on perceived levels of corruption in the procurement department in the judiciary of kenya, human resource factors influencing gender disparity in development of early childhood education teaching profession in kakamega east sub-county, kenya, nature and extent of women’s socioeconomic conflicts along lake victoria, kisumu county, kenya, intrinsic and extrinsic risk factors for nosocomial pressure injury among hospitalized adults at a tertiary hospital in western kenya, effect of external debt on performance of the agricultural sector in kenya, influence of sexual attitudes on the education participation of secondary school girls in homa bay county, kenya, effect of performance planning on employee productivity in broadcast media outlets in bungoma and busia counties, kenya, effects of debtors’ management on financial performance of savings and credit cooperative organizations in nandi county, kenya, customization and its influence on the performance of international courier firms in kenya, co-text and implicature in bi. msafwari’s topical discussions, factors influencing the management of musculoskeletal pain among children with sickle cell disease in western kenya, effect of mobile banking on financial inclusion among small-medium enterprises in kakamega county, kenya, influence of electronic payment on business efficiency in petrol stations in nyanza region, kenya, freshwater aquaculture and household performance in busia county, kenya, pre-service teachers' perceptions of identity formation in ghana, effects of virtual laboratory-based instruction on the frequency of use of experiment as a pedagogical approach in teaching and learning of physics in secondary schools in kenya, leveraging on virtual laboratory-based instruction to achieve active classroom interaction in teaching and learning of physics in secondary schools in kenya, factors influencing the use of mobile phone-enabled services in accessing agricultural information by smallholder farmers in bungoma county, kenya, the socio-economic life of fisher communities living on the migingo island of lake victoria, moderating effect of organizational culture on the relationship between quality of work life and organizational commitment amongst nurses in public hospitals in bungoma county, communicating off-record politeness by use of non-usual stress in kiswahili language, exploring the influence of domestic violence exposure on pupils' behavior in kenyan public primary schools, effect of mobile phone-based digital technologies on the performance of refugee women-led micro, small and medium enterprises in kakuma refugee camp, effect of procurement planning on procurement function performance in kakamega county government, credit risk management on financial performance of selected microfinance institutions, effect of quantity environmental reporting disclosure on financial performance of manufacturing firms listed in the nairobi securities exchange, enhancing national cohesion through primary social studies learning experiences in kenya, effectiveness of biometric modalities employed in management of identity based conflicts in nairobi county, kenya, single parenting and children’s academic performance in primary schools in embu east sub county, kenya, socio economic determinants and effective menstrual management among primary school girls in rural schools in nyatike sub county, trends in progression in agriculture career among students in tertiary institutions of kakamega and bungoma counties, kenya, influence of instructional resource utilization on kiswahili performance in secondary schools in vihiga county, kenya, nature and extent of deviant behaviour among secondary students in public schools within bungoma county, kenya, social and economic influence of abanyala clan politics on resource distribution, the significance of primary social studies content organization in fostering knowledge, skills, and values for cohesion and integration in kenya, a history of the isukha-idakho bullfighting in western kenya, adoption of agroforestry as a climate smart agriculture practice among smallholder farmers in kakamega county, kenya agroforestry, assessment of the influence of teaching methods on learners preparedness for primary education in kenya, assessing the socio-economic impact of the chinese transfer of values, skills, and technology in infrastructural development in kenya, efficacy of learning materials on learners’ readiness for primary education in kenya, appreciating the theoretical perspectives of the rwandan genocide: implications for conflict resolution practice, social impact of dental caries on adult patients in bungoma county, kenya, application of the vector autoregressive model incorporating new measurements using the bayesian approach, strategies for mitigating flood risks in western region, kenya, impacts of sustainable agricultural practices on food security, nutrition, and poverty among smallholder maize farmers in morogoro region, tanzania, diagnostic assessment knowledge and practices among secondary school mathematics teachers in dodoma, tanzania, types of urban vulnerabilities affecting households in informal settlements of kisumu county, assessing the prospects and challenges of online learning in kenyan public universities: a case study of masinde muliro university of science and technology, utilization of interpersonal communication channels in the uptake of cervical cancer screening at moi teaching and referral hospital, kenya, supplier risk management practices and performance of supply chain in the health sector in kenya, influence of capital risk on financial performance of microfinance institutions in kenya.

AJOL is a Non Profit Organisation that cannot function without donations. AJOL and the millions of African and international researchers who rely on our free services are deeply grateful for your contribution. AJOL is annually audited and was also independently assessed in 2019 by E&Y.

Your donation is guaranteed to directly contribute to Africans sharing their research output with a global readership.

• For annual AJOL Supporter contributions, please view our Supporters page.

Journal Identifiers

Investing with Purpose: The Role of CSR in Enhancing Chinese Firms’ Performance in Japan

• Published: 09 April 2024

Cite this article

• Xu Chen 1 ,
• Xuyang Dong 2 &
• Chao Ma 3  

This research paper delves into the intricate dynamics of Chinese firms’ foreign direct investment (FDI) in Japan, shedding light on the “China model” of foreign investment. While previous studies have extensively explored Chinese FDI in resource-rich sectors, this paper focuses on the unique challenges and opportunities in a developed country like Japan, which shares similar technological capabilities with China. In particular, we investigate the role of corporate social responsibility (CSR) in mediating the efficiency of Chinese companies’ investments in Japan. As China’s economy continues to grow globally, understanding the effectiveness of its FDI activities has become increasingly important. This study employs a comprehensive approach, drawing insights from various academic perspectives, including digital finance, product market competition, education systems, energy intensity, logistical capability, and green credit policies, to provide a holistic view of Chinese FDI in Japan. Our findings emphasize the pivotal role of CSR in enhancing investment efficiency and mitigating challenges faced by Chinese firms in Japan. We highlight how CSR positively influences organizational trust, credibility, and risk management, ultimately improving investment effectiveness. Additionally, resource-based theory studies reveal the connection between CSR, technological innovation, and corporate growth, further enhancing investment efficiency. This research, grounded in stakeholder and information asymmetry theories, utilizes robust statistical methods and a rich dataset to establish and validate hypotheses. The results underscore the significance of CSR as a mediating factor and its positive impact on Chinese companies’ FDI efficiency in Japan. This study aligns with the principles of the knowledge economy and provides practical insights for policymakers in China and Japan. Both nations can foster a conducive environment for sustainable and mutually beneficial international collaborations in the global marketplace by promoting CSR practices and recognizing their influence on investment efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Data Availability

The data used to support the findings of this study are included within the article.

Abad-Segura, E., Cortés-García, F. J., & Belmonte-Ureña, L. J. (2019). The sustainable approach to corporate social responsibility: A global analysis and future trends. Sustainability, 11 (19), 5382.

Article   Google Scholar  

Alam, S. S., & Islam, K. Z. (2021). Examining the role of environmental corporate social responsibility in building green corporate image and green competitive advantage. International Journal of Corporate Social Responsibility, 6 (1), 8.

Bardy, R., Drew, S., & Kennedy, T. F. (2012). Foreign investment and ethics: How to contribute to social responsibility by doing business in less-developed countries. Journal of Business Ethics, 106 , 267–282.

Benlemlih, M., & Bitar, M. (2018). Corporate social responsibility and investment efficiency. Journal of Business Ethics, 148 , 647–671.

Bergh, D. D., Ketchen, D. J., Jr., Orlandi, I., Heugens, P. P., & Boyd, B. K. (2019). Information asymmetry in management research: Past accomplishments and future opportunities. Journal of Management, 45 (1), 122–158.

Blowfield, M., & Frynas, J. G. (2005). Editorial setting new agendas: Critical perspectives on Corporate Social Responsibility in the developing world. International Affairs, 81 (3), 499–513.

Boubaker, S., Dang, V. A., & Sassi, S. (2022). Competitive pressure and firm investment efficiency: Evidence from corporate employment decisions. European Financial Management, 28 (1), 113–161.

Bouichou, S. I., Wang, L., & Zulfiqar, S. (2022). How corporate social responsibility boosts corporate financial and non-financial performance: The moderating role of ethical leadership. Frontiers in Psychology, 13 , 871334.

Brammer, S., Millington, A., & Rayton, B. (2007). The contribution of corporate social responsibility to organizational commitment. The International Journal of Human Resource Management, 18 (10), 1701–1719.

Branco, M. C., & Rodrigues, L. L. (2006). Corporate social responsibility and resource-based perspectives. Journal of Business Ethics, 69 , 111–132.

Buckley, P. J. (2018). Towards a theoretically-based global foreign direct investment policy regime. Journal of International Business Policy, 1 , 184–207.

Cao, J., Liang, H., & Zhan, X. (2019). Peer effects of corporate social responsibility. Management Science, 65 (12), 5487–5503.

Carroll, A. B., & Shabana, K. M. (2010). The business case for corporate social responsibility: A review of concepts, research and practice. International Journal of Management Reviews, 12 (1), 85–105.

Chen, S., & Ji, Y. (2022). Do corporate social responsibility categories distinctly influence innovation? A Resource-Based Theory Perspective. Sustainability, 14 (6), 3154.

Google Scholar  

Cheynel, E., & Levine, C. B. (2020). Public disclosures and information asymmetry: A theory of the mosaic. The Accounting Review, 95 (1), 79–99.

Cho, S. J., Chung, C. Y., & Young, J. (2019). Study on the relationship between CSR and financial performance. Sustainability, 11 (2), 343.

Cook, K. A., Romi, A. M., Sánchez, D., & Sánchez, J. M. (2019). The influence of corporate social responsibility on investment efficiency and innovation. Journal of Business Finance & Accounting, 46 (3–4), 494–537.

Corlu, C. G., Goyal, A., Lopez-Lopez, D., Torre, R. D. L., & Juan, A. A. (2021). Ranking enterprise reputation in the digital age: A survey of traditional methods and the need for more agile approaches. International Journal of Data Analysis Techniques and Strategies, 13 (4), 265–290.

Dhar, B. K., Sarkar, S. M., & Ayittey, F. K. (2022). Impact of social responsibility disclosure between implementation of green accounting and sustainable development: A study on heavily polluting companies in Bangladesh. Corporate Social Responsibility and Environmental Management, 29 (1), 71–78.

Djalilov, K. (2022). A contingent resource-based perspective on corporate social responsibility and competitive advantage: A focus on transition countries. Business Ethics and Leadership, 6 (1), 92–108.

Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., ... & Williams, M. D. (2021). Artificial intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International Journal of Information Management , 57 , 101994.

Eberle, D., Berens, G., & Li, T. (2013). The impact of interactive corporate social responsibility communication on corporate reputation. Journal of Business Ethics, 118 , 731–746.

Eweje, G., & Sakaki, M. (2015). CSR in Japanese companies: Perspectives from managers. Business Strategy and the Environment, 24 (7), 678–687.

Falck, O., & Heblich, S. (2007). Corporate social responsibility: Doing well by doing good. Business Horizons, 50 (3), 247–254.

Fazzari, S. M., & Variato, A. M. (1994). Asymmetric information and Keynesian theories of investment. Journal of Post Keynesian Economics , 16 (3), 351–369.

Fukukawa, K., & Teramoto, Y. (2009). Understanding Japanese CSR: The reflections of managers in the field of global operations. Journal of Business Ethics, 85 , 133–146.

Gallego-Álvarez, I., Manuel Prado-Lorenzo, J., & García-Sánchez, I. M. (2011). Corporate social responsibility and innovation: A resource-based theory. Management Decision, 49 (10), 1709–1727.

Gao, G. Y., Wang, D. T., & Che, Y. (2018). Impact of historical conflict on FDI location and performance: Japanese investment in China. Journal of International Business Studies, 49 , 1060–1080.

Gazzola, P., & Mella, P. (2012). Corporate performance and corporate social responsibility (CSR). A necessary choice? Economia Aziendale Online-, 3 , 1–22.

George, N. A., Aboobaker, N., & Edward, M. (2020). Corporate social responsibility and organizational commitment: Effects of CSR attitude, organizational trust and identification. Society and Business Review, 15 (3), 255–272.

George, N. A., Aboobaker, N., & Edward, M. (2021). Corporate social responsibility, organizational trust and commitment: A moderated mediation model. Personnel Review, 50 (4), 1093–1111.

Goyal, A. (2006). Corporate social responsibility as a signalling device for foreign direct investment. International Journal of the Economics of Business, 13 (1), 145–163.

Guan, Y., Zhai, Z., Wang, Y., Wu, D., Yu, L., & Lei, Z. (2022). Foreign direct investment, environmental regulation, and haze pollution: Empirical evidence from China. Environmental Science and Pollution Research, 29 (18), 27571–27584.

Gunby, P., Jin, Y., & Reed, W. R. (2017). Did FDI really cause Chinese economic growth? A meta-analysis. World Development, 90 , 242–255.

He, F., Liu, R., & Chen, L. (2020). Will fulfilling environmental responsibility improve corporate economic benefits?—From the perspective of stakeholders. Journal of Beijing Institute of Technology (social Sciences Edition), 22 (6), 32–42.

Heslin, P. A., & Ochoa, J. D. (2008). Understanding and developing strategic corporate social responsibility. Organizational Dynamics, 37 , 125–144.

Huang, Z., Tao, Y., Luo, X., Ye, Y., & Lei, T. (2023). Regional digital finance and corporate investment efficiency in China. Applied Economics, 55 (43), 5115–5134.

Jamali, D., Safieddine, A. M., & Rabbath, M. (2008). Corporate governance and corporate social responsibility synergies and interrelationships. Corporate Governance: An International Review, 16 (5), 443–459.

Knoerich, J. (2016). Has outward foreign direct investment contributed to the development of the Chinese economy. Transnational Corporations, 23 (2), 1–48.

Kölbel, J. F., & Busch, T. (2021). Signaling legitimacy across institutional contexts—The intermediary role of corporate social responsibility rating agencies. Global Strategy Journal, 11 (2), 304–328.

Kolk, A., & Pinkse, J. (2006). Stakeholder mismanagement and corporate social responsibility crises. European Management Journal, 24 (1), 59–72.

Kong, Q., Peng, D., Ruijia, Z., & Wong, Z. (2021). Resource misallocation, production efficiency and outward foreign direct investment decisions of Chinese enterprises. Research in International Business and Finance, 55 , 101343.

Krishnaswami, S., & Subramaniam, V. (1999). Information asymmetry, valuation, and the corporate spin-off decision. Journal of Financial Economics, 53 (1), 73–112.

Krukowska, M. (2014). Determinants of corporate social responsibility in Japanese companies. Management and Business Administration. Central Europe, 127 (4), 39–57.

Li, G., Li, L., Choi, T. M., & Sethi, S. P. (2020). Green supply chain management in Chinese firms: Innovative measures and the moderating role of quick response technology. Journal of Operations Management, 66 (7–8), 958–988.

Li, Y., Wang, J., Mu, Z., & Li, L. (2023). The impact of corporate environmental responsibility on green technological innovation: A nonlinear model including mediate effects and moderate effects. Economic Analysis and Policy, 80 , 754–769.

Lin, Y. E., Li, Y. W., Cheng, T. Y., & Lam, K. (2021). Corporate social responsibility and investment efficiency: Does business strategy matter? International Review of Financial Analysis, 73 , 101585.

Liu, M., Marshall, A., & McColgan, P. (2021). Foreign direct investments: The role of corporate social responsibility. Journal of Multinational Financial Management, 59 , 100663.

Lo, D., Hong, F., & Li, G. (2016). Assessing the role of inward foreign direct investment in Chinese economic development, 1990–2007: Towards a synthesis of alternative views. Structural Change and Economic Dynamics, 37 , 107–120.

Lock, I., & Seele, P. (2016). The credibility of CSR (corporate social responsibility) reports in Europe. Evidence from a quantitative content analysis in 11 countries. Journal of Cleaner Production, 122 , 186–200.

Lojpur, A., & Draskovic, V. (2013). The institutional context of corporate governance and corporate social responsibility. Montenegrin Journal of Economics, 9 (1), 27.

Lou, Z., Chen, S., Yin, W., Zhang, C., & Yu, X. (2022). Economic policy uncertainty and firm innovation: Evidence from a risk-taking perspective. International Review of Economics & Finance, 77 , 78–96.

Madueno, J. H., Jorge, M. L., Conesa, I. M., & Martínez-Martínez, D. (2016). Relationship between corporate social responsibility and competitive performance in Spanish SMEs: Empirical evidence from a stakeholders’ perspective. BRQ Business Research Quarterly, 19 (1), 55–72.

Maskrey, A., Jain, G., & Lavell, A. (2023). The social construction of systemic risk: towards an actionable framework for risk governance. Disaster Prevention and Managment , 32(1), 4-26

Meng, M., Tao, Q., & Lei, J. (2019). Corporate social responsibility and corporate growth: The intermediary effect of technological innovation. Research and Development Management, 31 , 27–37.

Menshawy, I. M., Basiruddin, R., Mohd-Zamil, N. A., & Hussainey, K. (2023). Strive towards investment efficiency among Egyptian companies: Do board characteristics and information asymmetry matter? International Journal of Finance & Economics, 28 (3), 2382–2403.

Miningou, E. W., & Tapsoba, S. J. (2020). Education systems and foreign direct investment: Does external efficiency matter? Journal of Applied Economics, 23 (1), 583–599.

Moran, T. (2011). Foreign direct investment and development: Launching a second generation of policy research: Avoiding the mistakes of the first, reevaluating policies for developed and developing countries . Columbia University Press.

Naseem, T., Shahzad, F., Asim, G. A., Rehman, I. U., & Nawaz, F. (2020). Corporate social responsibility engagement and firm performance in Asia Pacific: The role of enterprise risk management. Corporate Social Responsibility and Environmental Management, 27 (2), 501–513.

Nguyen, T., Pham, T., Le, Q., & Bui, T. (2020). Impact of corporate social responsibility on organizational commitment through organizational trust and organizational identification. Management Science Letters, 10 (14), 3453–3462.

Nurlia, N., Daud, I., & Rosadi, M. E. (2023). AI implementation impact on workforce productivity: The role of ai training and organizational adaptation. Escalate: Economics and Business Journal, 1 (01), 01–13.

Ortiz-de-Mandojana, N., & Bansal, P. (2016). The long-term benefits of organizational resilience through sustainable business practices. Strategic Management Journal, 37 (8), 1615–1631.

Park, Y., Park, Y., Hong, P. C., & Yang, S. (2017). Clarity of CSR orientation and firm performance: Case of Japanese SMEs. Benchmarking: An International Journal, 24 (6), 1581–1596.

Peng, G. Z., & Beamish, P. W. (2008). The effect of national corporate responsibility environment on Japanese foreign direct investment. Journal of Business Ethics, 80 , 677–695.

Rehman, I. U., Shahzad, F., Latif, K. F., Nawab, N., Rashid, A., & Hyder, S. (2021). Does corporate social responsibility mediate the influence of national culture on investment inefficiency? Firm-level evidence from Asia Pacific. International Journal of Finance & Economics, 26 (3), 3484–3503.

Schaede, U. (2020). The business reinvention of Japan: How to make sense of the new Japan and why it matters . Stanford University Press.

Book   Google Scholar  

Scholtens, B. (2006). Finance as a driver of corporate social responsibility. Journal of Business Ethics, 68 , 19–33.

Smith, A. D. (2007). Making the case for the competitive advantage of corporate social responsibility. Business Strategy Series, 8 (3), 186–195.

Soh, K. L., Wong, W. P., & Tang, C. F. (2021). The role of institutions at the nexus of logistic performance and foreign direct investment in Asia. The Asian Journal of Shipping and Logistics, 37 (2), 165–173.

Su, Y., & Liu, Z. (2016). The impact of foreign direct investment and human capital on economic growth: Evidence from Chinese cities. China Economic Review, 37 , 97–109.

Sun, C., Guo, Z., & Wang, Z. (2023). Outward foreign direct investment and energy intensity: Evidence from the listed companies in China. Environmental Science and Pollution Research, 30 (10), 27056–27072.

Sun, W., & Cui, K. (2014). Linking corporate social responsibility to firm default risk. European Management Journal, 32 (2), 275–287.

Suto, M., & Takehara, H. (2016). The link between corporate social performance and financial performance: Empirical evidence from Japanese firms. International Journal of Corporate Strategy and Social Responsibility, 1 (1), 4–25.

Suzuki, K., Tanimoto, K., & Kokko, A. (2010). Does foreign investment matter? Effects of foreign investment on the institutionalisation of corporate social responsibility by Japanese firms. Asian Business & Management, 9 , 379–400.

Suzuki, S., Sasaki, H., & Davis, S. (2021). Corporate social responsibility in Japan: Responsible business in a changing Japan. Current Global Practices of Corporate Social Responsibility: In the Era of Sustainable Development Goals , 1 (1), 745–775.

Tokoro, N. (2007). Stakeholders and corporate social responsibility (CSR): A new perspective on the structure of relationships. Asian Business & Management, 6 , 143–162.

Uduji, J. I., & Okolo, E. N. (2023). Ecotourism for transformative and youth development in sub-Saharan Africa: the role of corporate social responsibility in Nigeria's oil host communities. Journal of Tourism and Cultural Change , 21 (6), 629–656.

Uribe, D. F., Ortiz-Marcos, I., & Uruburu, Á. (2018). What is going on with stakeholder theory in project management literature? A Symbiotic Relationship for Sustainability. Sustainability, 10 (4), 1300.

Wang, D., Zhu, W., Zhang, C., Li, H., & Wu, Y. (2020). Stakeholder symbiosis in the context of corporate social responsibility. Symmetry, 12 (11), 1897.

Wokutch, R. E. (1990). Corporate social responsibility Japanese style. Academy of Management Perspectives, 4 (2), 56–74.

Wolfolds, S. E., & Siegel, J. (2019). Misaccounting for endogeneity: The peril of relying on the Heckman two-step method without a valid instrument. Strategic Management Journal, 40 (3), 432–462.

Xuemei, X., Ruoyi, W., & Jiage, H. (2020). Green process innovation and corporate performance in the context of government’s financial incentive: An empirical study based on content analysis. Management Review, 32 (5), 109.

Ye, J., & Dela, E. (2023). The effect of green investment and green financing on sustainable business performance of foreign chemical industries operating in Indonesia: The mediating role of corporate social responsibility. Sustainability, 15 (14), 11218.

Yuan, B., & Cao, X. (2022). Do corporate social responsibility practices contribute to green innovation? The mediating role of green dynamic capability. Technology in Society, 68 , 101868.

Zahra, S. A., & Garvis, D. M. (2000). International corporate entrepreneurship and firm performance: The moderating effect of international environmental hostility. Journal of Business Venturing, 15 (5–6), 469–492.

Zhang, J., Luo, Y., & Ding, X. (2022). Can green credit policy improve the overseas investment efficiency of enterprises in China? Journal of Cleaner Production, 340 , 130785.

Zhang, Y., Zhang, J., & Cheng, Z. (2021). Stock market liberalization and corporate green innovation: Evidence from China. International Journal of Environmental Research and Public Health, 18 (7), 3412.

Zhao, J. (2021). Reimagining corporate social responsibility in the era of COVID-19: Embedding resilience and promoting corporate social competence. Sustainability, 13 (12), 6548.

Zhu, Q., & Lai, K. H. (2019). Enhancing supply chain operations with extended corporate social responsibility practices by multinational enterprises: Social capital perspective from Chinese suppliers. International Journal of Production Economics, 213 , 1–12.

Zimon, G., Arianpoor, A., & Salehi, M. (2022). Sustainability reporting and corporate reputation: The moderating effect of CEO opportunistic behavior. Sustainability, 14 (3), 1257.

Download references

This article was supported by the Scientific Research Program Funded by Shaanxi Provincial Education Department: Research on China’s National Image Reported by Japanese Media in the Post-epidemic Era (Program No. 21JK0289).

Author information

Authors and affiliations.

School of Foreign Languages, Xi’an Shiyou University, Xi’an, Shaanxi, China

College of Foreign Studies, Jiaxing University, Jiaxing, Zhejiang, China

Xuyang Dong

Pan Tianshou College of Architecture and Art Design, Ningbo University, Ningbo, China

You can also search for this author in PubMed   Google Scholar

Contributions

Xu Chen is responsible for writing the paper, building the framework, logical analysis, and data analysis. Xuyang Dong is responsible for coordinating and planning research papers, writing papers, proofreading language, and analyzing data. Chao Ma is responsible for date analysis the paper.

Corresponding authors

Correspondence to Xuyang Dong or Chao Ma .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on Innovation Management in Asia

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Chen, X., Dong, X. & Ma, C. Investing with Purpose: The Role of CSR in Enhancing Chinese Firms’ Performance in Japan. J Knowl Econ (2024). https://doi.org/10.1007/s13132-024-01875-3

Download citation

Received : 27 September 2023

Accepted : 17 February 2024

Published : 09 April 2024

DOI : https://doi.org/10.1007/s13132-024-01875-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Corporate social responsibility (CSR)
• Foreign direct investment (FDI) efficiency
• Chinese companies
• Richardson’s investment expectation mode
• Descriptive statistics
• Socially responsible practices

Advertisement

• Find a journal
• Publish with us
• Track your research