methodology for a literature review

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved September 9, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

• University of Texas Libraries

Literature Reviews

Steps in the literature review process.

• What is a literature review?
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support
• Artificial Intelligence (AI) Tools
• You may need to some exploratory searching of the literature to get a sense of scope, to determine whether you need to narrow or broaden your focus
• Identify databases that provide the most relevant sources, and identify relevant terms (controlled vocabularies) to add to your search strategy
• Finalize your research question
• Think about relevant dates, geographies (and languages), methods, and conflicting points of view
• Conduct searches in the published literature via the identified databases
• Check to see if this topic has been covered in other discipline's databases
• Examine the citations of on-point articles for keywords, authors, and previous research (via references) and cited reference searching.
• Save your search results in a citation management tool (such as Zotero, Mendeley or EndNote)
• De-duplicate your search results
• Make sure that you've found the seminal pieces -- they have been cited many times, and their work is considered foundational 
• Check with your professor or a librarian to make sure your search has been comprehensive
• Evaluate the strengths and weaknesses of individual sources and evaluate for bias, methodologies, and thoroughness
• Group your results in to an organizational structure that will support why your research needs to be done, or that provides the answer to your research question  
• Develop your conclusions
• Are there gaps in the literature?
• Where has significant research taken place, and who has done it?
• Is there consensus or debate on this topic?
• Which methodological approaches work best?
• For example: Background, Current Practices, Critics and Proponents, Where/How this study will fit in 
• Organize your citations and focus on your research question and pertinent studies
• Compile your bibliography

Note: The first four steps are the best points at which to contact a librarian. Your librarian can help you determine the best databases to use for your topic, assess scope, and formulate a search strategy.

Videos Tutorials about Literature Reviews

This 4.5 minute video from Academic Education Materials has a Creative Commons License and a British narrator.

Recommended Reading

• Last Updated: Aug 26, 2024 5:59 AM
• URL: https://guides.lib.utexas.edu/literaturereviews

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

Prevent plagiarism, run a free check.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 9 September 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley Open Access Collection

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure  1 . A list of excluded studies with reasons can be found in Appendix B .

Study selection flowchart

3.1. Characteristics of included reviews

Table  1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n  = 3), literature search ( n  = 3), study selection ( n  = 2), data extraction ( n  = 1), and RoB assessment ( n  = 2) (Table  2 ).

Summary of findings from review evaluating systematic review methods

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n  = 4), Hopewell et al. 20 and Crumley et al. 19 ( n  = 30), and Robson et al. 14 and Morissette et al. 23 ( n  = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table  2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration  or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Jul 15, 2024 10:34 AM
• URL: https://guides.lib.udel.edu/researchmethods

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

What is the purpose of literature review , a. habitat loss and species extinction: , b. range shifts and phenological changes: , c. ocean acidification and coral reefs: , d. adaptive strategies and conservation efforts: .

• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• How to write a literature review faster with Paperpal? 

Frequently asked questions 

What is a literature review .

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

1. Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 

2. Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field.

Find academic papers related to your research topic faster. Try Research on Paperpal

3. Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 

4. Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 

5. Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 

6. Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example 

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:  

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

Strengthen your literature review with factual insights. Try Research on Paperpal for free!

How to write a good literature review 

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Write and Cite as yo u go with Paperpal Research. Start now for free!

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review 

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:  

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:  

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:  

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:  

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:  

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:  

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

Whether you’re exploring a new research field or finding new angles to develop an existing topic, sifting through hundreds of papers can take more time than you have to spare. But what if you could find science-backed insights with verified citations in seconds? That’s the power of Paperpal’s new Research feature!  

How to write a literature review faster with Paperpal?  

Paperpal, an AI writing assistant, integrates powerful academic search capabilities within its writing platform. With the Research | Cite feature, you get 100% factual insights, with citations backed by 250M+ verified research articles, directly within your writing interface. It also allows you auto-cite references in 10,000+ styles and save relevant references in your Citation Library. By eliminating the need to switch tabs to find answers to all your research questions, Paperpal saves time and helps you stay focused on your writing.   

Here’s how to use the Research feature:  

• Ask a question: Get started with a new document on paperpal.com. Click on the “Research | Cite” feature and type your question in plain English. Paperpal will scour over 250 million research articles, including conference papers and preprints, to provide you with accurate insights and citations. 

• Review and Save: Paperpal summarizes the information, while citing sources and listing relevant reads. You can quickly scan the results to identify relevant references and save these directly to your built-in citations library for later access. 
• Cite with Confidence: Paperpal makes it easy to incorporate relevant citations and references in 10,000+ styles into your writing, ensuring your arguments are well-supported by credible sources. This translates to a polished, well-researched literature review. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a good literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. By combining effortless research with an easy citation process, Paperpal Research streamlines the literature review process and empowers you to write faster and with more confidence. Try Paperpal Research now and see for yourself.  

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 22+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• How Long Should a Chapter Be?
• How to Use Paperpal to Generate Emails & Cover Letters?

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, academic integrity vs academic dishonesty: types & examples, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , the ai revolution: authors’ role in upholding academic..., the future of academia: how ai tools are..., how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide).

How to Write a Literature Review: Six Steps to Get You from Start to Finish

Writing-a-literature-review-six-steps-to-get-you-from-start-to-finish.

Tanya Golash-Boza, Associate Professor of Sociology, University of California

February 03, 2022

Writing a literature review is often the most daunting part of writing an article, book, thesis, or dissertation. “The literature” seems (and often is) massive. I have found it helpful to be as systematic as possible when completing this gargantuan task.

Sonja Foss and William Walters* describe an efficient and effective way of writing a literature review. Their system provides an excellent guide for getting through the massive amounts of literature for any purpose: in a dissertation, an M.A. thesis, or preparing a research article for publication  in any field of study. Below is a  summary of the steps they outline as well as a step-by-step method for writing a literature review.

How to Write a Literature Review

Step One: Decide on your areas of research:

Before you begin to search for articles or books, decide beforehand what areas you are going to research. Make sure that you only get articles and books in those areas, even if you come across fascinating books in other areas. A literature review I am currently working on, for example, explores barriers to higher education for undocumented students.

Step Two: Search for the literature:

Conduct a comprehensive bibliographic search of books and articles in your area. Read the abstracts online and download and/or print those articles that pertain to your area of research. Find books in the library that are relevant and check them out. Set a specific time frame for how long you will search. It should not take more than two or three dedicated sessions.

Step Three: Find relevant excerpts in your books and articles:

Skim the contents of each book and article and look specifically for these five things:

1. Claims, conclusions, and findings about the constructs you are investigating

2. Definitions of terms

3. Calls for follow-up studies relevant to your project

4. Gaps you notice in the literature

5. Disagreement about the constructs you are investigating

When you find any of these five things, type the relevant excerpt directly into a Word document. Don’t summarize, as summarizing takes longer than simply typing the excerpt. Make sure to note the name of the author and the page number following each excerpt. Do this for each article and book that you have in your stack of literature. When you are done, print out your excerpts.

Step Four: Code the literature:

Get out a pair of scissors and cut each excerpt out. Now, sort the pieces of paper into similar topics. Figure out what the main themes are. Place each excerpt into a themed pile. Make sure each note goes into a pile. If there are excerpts that you can’t figure out where they belong, separate those and go over them again at the end to see if you need new categories. When you finish, place each stack of notes into an envelope labeled with the name of the theme.

Step Five: Create Your Conceptual Schema:

Type, in large font, the name of each of your coded themes. Print this out, and cut the titles into individual slips of paper. Take the slips of paper to a table or large workspace and figure out the best way to organize them. Are there ideas that go together or that are in dialogue with each other? Are there ideas that contradict each other? Move around the slips of paper until you come up with a way of organizing the codes that makes sense. Write the conceptual schema down before you forget or someone cleans up your slips of paper.

Step Six: Begin to Write Your Literature Review:

Choose any section of your conceptual schema to begin with. You can begin anywhere, because you already know the order. Find the envelope with the excerpts in them and lay them on the table in front of you. Figure out a mini-conceptual schema based on that theme by grouping together those excerpts that say the same thing. Use that mini-conceptual schema to write up your literature review based on the excerpts that you have in front of you. Don’t forget to include the citations as you write, so as not to lose track of who said what. Repeat this for each section of your literature review.

Once you complete these six steps, you will have a complete draft of your literature review. The great thing about this process is that it breaks down into manageable steps something that seems enormous: writing a literature review.

I think that Foss and Walter’s system for writing the literature review is ideal for a dissertation, because a Ph.D. candidate has already read widely in his or her field through graduate seminars and comprehensive exams.

It may be more challenging for M.A. students, unless you are already familiar with the literature. It is always hard to figure out how much you need to read for deep meaning, and how much you just need to know what others have said. That balance will depend on how much you already know.

For people writing literature reviews for articles or books, this system also could work, especially when you are writing in a field with which you are already familiar. The mere fact of having a system can make the literature review seem much less daunting, so I recommend this system for anyone who feels overwhelmed by the prospect of writing a literature review.

*Destination Dissertation: A Traveler's Guide to a Done Dissertation

Image Credit/Source: Goldmund Lukic/Getty Images

Watch our Webinar to help you get published

Please enter your Email Address

Please enter valid email address

Please Enter your First Name

Please enter your Last Name

Please enter your Questions or Comments.

Please enter the Privacy

Please enter the Terms & Conditions

Leveraging user research to improve author guidelines at the Council of Science Editors Annual Meeting

How research content supports academic integrity

Finding time to publish as a medical student: 6 tips for Success

Software to Improve Reliability of Research Image Data: Wiley, Lumina, and Researchers at Harvard Medical School Work Together on Solutions

Driving Research Outcomes: Wiley Partners with CiteAb

ISBN, ISSN, DOI: what they are and how to find them

Image Collections for Medical Practitioners with TDS Health

How do you Discover Content?

Writing for Publication for Nurses (Mandarin Edition)

Get Published - Your How to Webinar

Related articles.

User Experience (UX) Research is the process of discovering and understanding user requirements, motivations, and behaviours 

Learn how Wiley partners with plagiarism detection services to support academic integrity around the world

Medical student Nicole Foley shares her top tips for writing and getting your work published.

Wiley and Lumina are working together to support the efforts of researchers at Harvard Medical School to develop and test new machine learning tools and artificial intelligence (AI) software that can

Learn more about our relationship with a company that helps scientists identify the right products to use in their research

What is ISBN? ISSN? DOI? Learn about some of the unique identifiers for book and journal content.

Learn how medical practitioners can easily access and search visual assets from our article portfolio

Explore free-to-use services that can help you discover new content

Watch this webinar to help you learn how to get published.

Finding time to publish as a medical student: 6 tips for success

How to Easily Access the Most Relevant Research: A Q&A With the Creator of Scitrus

Atypon launches Scitrus, a personalized web app that allows users to create a customized feed of the latest research.

FOR INDIVIDUALS

FOR INSTITUTIONS & BUSINESSES

WILEY NETWORK

ABOUT WILEY

Corporate Responsibility

Corporate Governance

Leadership Team

Cookie Preferences

Copyright @ 2000-2024  by John Wiley & Sons, Inc., or related companies. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.

Rights & Permissions

Privacy Policy

Terms of Use

Research Methods and Design

• Action Research
• Case Study Design

Literature Review

• Quantitative Research Methods
• Qualitative Research Methods
• Mixed Methods Study
• Indigenous Research and Ethics This link opens in a new window
• Identifying Empirical Research Articles This link opens in a new window
• Research Ethics and Quality
• Data Literacy
• Get Help with Writing Assignments

A literature review is a discussion of the literature (aka. the "research" or "scholarship") surrounding a certain topic. A good literature review doesn't simply summarize the existing material, but provides thoughtful synthesis and analysis. The purpose of a literature review is to orient your own work within an existing body of knowledge. A literature review may be written as a standalone piece or be included in a larger body of work.

You can read more about literature reviews, what they entail, and how to write one, using the resources below. 

Am I the only one struggling to write a literature review?

Dr. Zina O'Leary explains the misconceptions and struggles students often have with writing a literature review. She also provides step-by-step guidance on writing a persuasive literature review.

An Introduction to Literature Reviews

Dr. Eric Jensen, Professor of Sociology at the University of Warwick, and Dr. Charles Laurie, Director of Research at Verisk Maplecroft, explain how to write a literature review, and why researchers need to do so. Literature reviews can be stand-alone research or part of a larger project. They communicate the state of academic knowledge on a given topic, specifically detailing what is still unknown.

This is the first video in a whole series about literature reviews. You can find the rest of the series in our SAGE database, Research Methods:

Videos covering research methods and statistics

Identify Themes and Gaps in Literature (with real examples) | Scribbr

Finding connections between sources is key to organizing the arguments and structure of a good literature review. In this video, you'll learn how to identify themes, debates, and gaps between sources, using examples from real papers.

4 Tips for Writing a Literature Review's Intro, Body, and Conclusion | Scribbr

While each review will be unique in its structure--based on both the existing body of both literature and the overall goals of your own paper, dissertation, or research--this video from Scribbr does a good job simplifying the goals of writing a literature review for those who are new to the process. In this video, you’ll learn what to include in each section, as well as 4 tips for the main body illustrated with an example.

• Literature Review This chapter in SAGE's Encyclopedia of Research Design describes the types of literature reviews and scientific standards for conducting literature reviews.
• UNC Writing Center: Literature Reviews This handout from the Writing Center at UNC will explain what literature reviews are and offer insights into the form and construction of literature reviews in the humanities, social sciences, and sciences.
• Purdue OWL: Writing a Literature Review The overview of literature reviews comes from Purdue's Online Writing Lab. It explains the basic why, what, and how of writing a literature review.

Organizational Tools for Literature Reviews

One of the most daunting aspects of writing a literature review is organizing your research. There are a variety of strategies that you can use to help you in this task. We've highlighted just a few ways writers keep track of all that information! You can use a combination of these tools or come up with your own organizational process. The key is choosing something that works with your own learning style.

Citation Managers

Citation managers are great tools, in general, for organizing research, but can be especially helpful when writing a literature review. You can keep all of your research in one place, take notes, and organize your materials into different folders or categories. Read more about citations managers here:

• Manage Citations & Sources

Concept Mapping

Some writers use concept mapping (sometimes called flow or bubble charts or "mind maps") to help them visualize the ways in which the research they found connects.

There is no right or wrong way to make a concept map. There are a variety of online tools that can help you create a concept map or you can simply put pen to paper. To read more about concept mapping, take a look at the following help guides:

• Using Concept Maps From Williams College's guide, Literature Review: A Self-guided Tutorial

Synthesis Matrix

A synthesis matrix is is a chart you can use to help you organize your research into thematic categories. By organizing your research into a matrix, like the examples below, can help you visualize the ways in which your sources connect. 

• Walden University Writing Center: Literature Review Matrix Find a variety of literature review matrix examples and templates from Walden University.
• Writing A Literature Review and Using a Synthesis Matrix An example synthesis matrix created by NC State University Writing and Speaking Tutorial Service Tutors. If you would like a copy of this synthesis matrix in a different format, like a Word document, please ask a librarian. CC-BY-SA 3.0
• << Previous: Case Study Design
• Next: Quantitative Research Methods >>
• Last Updated: May 7, 2024 9:51 AM

CityU Home - CityU Catalog

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT) 

How to write the methods section of a systematic review

Home | Blog | How To | How to write the methods section of a systematic review

Covidence breaks down how to write a methods section

The methods section of your systematic review describes what you did, how you did it, and why. Readers need this information to interpret the results and conclusions of the review. Often, a lot of information needs to be distilled into just a few paragraphs. This can be a challenging task, but good preparation and the right tools will help you to set off in the right direction 🗺️🧭.

Systematic reviews are so-called because they are conducted in a way that is rigorous and replicable. So it’s important that these methods are reported in a way that is thorough, clear, and easy to navigate for the reader – whether that’s a patient, a healthcare worker, or a researcher. 

Like most things in a systematic review, the methods should be planned upfront and ideally described in detail in a project plan or protocol. Reviews of healthcare interventions follow the PRISMA guidelines for the minimum set of items to report in the methods section. But what else should be included? It’s a good idea to consider what readers will want to know about the review methods and whether the journal you’re planning to submit the work to has expectations on the reporting of methods. Finding out in advance will help you to plan what to include.

Describe what happened

While the research plan sets out what you intend to do, the methods section is a write-up of what actually happened. It’s not a simple case of rewriting the plan in the past tense – you will also need to discuss and justify deviations from the plan and describe the handling of issues that were unforeseen at the time the plan was written. For this reason, it is useful to make detailed notes before, during, and after the review is completed. Relying on memory alone risks losing valuable information and trawling through emails when the deadline is looming can be frustrating and time consuming! 

Keep it brief

The methods section should be succinct but include all the noteworthy information. This can be a difficult balance to achieve. A useful strategy is to aim for a brief description that signposts the reader to a separate section or sections of supporting information. This could include datasets, a flowchart to show what happened to the excluded studies, a collection of search strategies, and tables containing detailed information about the studies.This separation keeps the review short and simple while enabling the reader to drill down to the detail as needed. And if the methods follow a well-known or standard process, it might suffice to say so and give a reference, rather than describe the process at length. 

Follow a structure

A clear structure provides focus. Use of descriptive headings keeps the writing on track and helps the reader get to key information quickly. What should the structure of the methods section look like? As always, a lot depends on the type of review but it will certainly contain information relating to the following areas:

• Selection criteria ⭕
• Data collection and analysis 👩‍💻
• Study quality and risk of bias ⚖️

Let’s look at each of these in turn.

1. Selection criteria ⭕

The criteria for including and excluding studies are listed here. This includes detail about the types of studies, the types of participants, the types of interventions and the types of outcomes and how they were measured. 

2. Search 🕵🏾‍♀️

Comprehensive reporting of the search is important because this means it can be evaluated and replicated. The search strategies are included in the review, along with details of the databases searched. It’s also important to list any restrictions on the search (for example, language), describe how resources other than electronic databases were searched (for example,  non-indexed journals), and give the date that the searches were run. The PRISMA-S extension provides guidance on reporting literature searches. 

Systematic reviewer pro-tip:

 Copy and paste the search strategy to avoid introducing typos

3. Data collection and analysis 👩‍💻

This section describes:

• how studies were selected for inclusion in the review
• how study data were extracted from the study reports
• how study data were combined for analysis and synthesis

To describe how studies were selected for inclusion , review teams outline the screening process. Covidence uses reviewers’ decision data to automatically populate a PRISMA flow diagram for this purpose. Covidence can also calculate Cohen’s kappa to enable review teams to report the level of agreement among individual reviewers during screening.

To describe how study data were extracted from the study reports , reviewers outline the form that was used, any pilot-testing that was done, and the items that were extracted from the included studies. An important piece of information to include here is the process used to resolve conflict among the reviewers. Covidence’s data extraction tool saves reviewers’ comments and notes in the system as they work. This keeps the information in one place for easy retrieval ⚡.

To describe how study data were combined for analysis and synthesis, reviewers outline the type of synthesis (narrative or quantitative, for example), the methods for grouping data, the challenges that came up, and how these were dealt with. If the review includes a meta-analysis, it will detail how this was performed and how the treatment effects were measured.

4. Study quality and risk of bias ⚖️

Because the results of systematic reviews can be affected by many types of bias, reviewers make every effort to minimise it and to show the reader that the methods they used were appropriate. This section describes the methods used to assess study quality and an assessment of the risk of bias across a range of domains. 

Steps to assess the risk of bias in studies include looking at how study participants were assigned to treatment groups and whether patients and/or study assessors were blinded to the treatment given. Reviewers also report their assessment of the risk of bias due to missing outcome data, whether that is due to participant drop-out or non-reporting of the outcomes by the study authors.

Covidence’s default template for assessing study quality is Cochrane’s risk of bias tool but it is also possible to start from scratch and build a tool with a set of custom domains if you prefer.

Careful planning, clear writing, and a structured approach are key to a good methods section. A methodologist will be able to refer review teams to examples of good methods reporting in the literature. Covidence helps reviewers to screen references, extract data and complete risk of bias tables quickly and efficiently. Sign up for a free trial today!

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Top 5 Tips for High-Quality Systematic Review Data Extraction

Data extraction can be a complex step in the systematic review process. Here are 5 top tips from our experts to help prepare and achieve high quality data extraction.

How to get through study quality assessment Systematic Review

Find out 5 tops tips to conducting quality assessment and why it’s an important step in the systematic review process.

How to extract study data for your systematic review

Learn the basic process and some tips to build data extraction forms for your systematic review with Covidence.

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers: 

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information. 

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Jul 8, 2024 3:13 PM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

Which review is that? A guide to review types

• Which review is that?
• Review Comparison Chart
• Decision Tool
• Critical Review
• Integrative Review
• Narrative Review
• State of the Art Review
• Narrative Summary
• Systematic Review
• Meta-analysis
• Comparative Effectiveness Review
• Diagnostic Systematic Review
• Network Meta-analysis
• Prognostic Review
• Psychometric Review
• Review of Economic Evaluations
• Systematic Review of Epidemiology Studies
• Living Systematic Reviews
• Umbrella Review
• Review of Reviews
• Rapid Review
• Rapid Evidence Assessment
• Rapid Realist Review
• Qualitative Evidence Synthesis
• Qualitative Interpretive Meta-synthesis
• Qualitative Meta-synthesis
• Qualitative Research Synthesis
• Framework Synthesis - Best-fit Framework Synthesis
• Meta-aggregation
• Meta-ethnography
• Meta-interpretation
• Meta-narrative Review
• Meta-summary
• Thematic Synthesis
• Mixed Methods Synthesis
• Narrative Synthesis
• Bayesian Meta-analysis
• EPPI-Centre Review
• Critical Interpretive Synthesis
• Realist Synthesis - Realist Review
• Scoping Review
• Mapping Review
• Systematised Review
• Concept Synthesis
• Expert Opinion - Policy Review
• Technology Assessment Review

Methodological Review

• Systematic Search and Review

A methodological review is a type of systematic secondary research (i.e., research synthesis) which focuses on summarising the state-of-the-art methodological practices of research in a substantive field or topic" (Chong et al, 2021).

Methodological reviews "can be performed to examine any methodological issues relating to the design, conduct and review of research studies and also evidence syntheses". Munn et al, 2018)

Further Reading/Resources

Clarke, M., Oxman, A. D., Paulsen, E., Higgins, J. P. T., & Green, S. (2011). Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Cochrane Handbook for systematic reviews of interventions . Full Text PDF

Aguinis, H., Ramani, R. S., & Alabduljader, N. (2023). Best-Practice Recommendations for Producers, Evaluators, and Users of Methodological Literature Reviews. Organizational Research Methods, 26(1), 46-76. https://doi.org/10.1177/1094428120943281 Full Text

Jha, C. K., & Kolekar, M. H. (2021). Electrocardiogram data compression techniques for cardiac healthcare systems: A methodological review. IRBM . Full Text

References Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC medical research methodology , 18 (1), 1-9. Full Text Chong, S. W., & Reinders, H. (2021). A methodological review of qualitative research syntheses in CALL: The state-of-the-art. System , 103 , 102646. Full Text

• << Previous: Technology Assessment Review
• Next: Systematic Search and Review >>
• Last Updated: Aug 19, 2024 1:08 PM
• URL: https://unimelb.libguides.com/whichreview

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

The state of the art of digital twins in health—a quick review of the literature.

1. Introduction

2. literature review, 2.1. digital twins, 2.2. digital health, 2.3. healthcare, 3. methodology, 5. discussion, 5.1. axis 1: use of digital twins for virtual representation of biological structures, 5.2. axis 2: use of digital twins to improve healthcare processes (personalized care), 5.3. axis 3: use of digital twins to depict healthcare structures and improve operational efficiency, 5.4. axis 4: use of digital twins for the development of medicines and health devices.

• Data Integration: Healthcare management involves a vast amount of information and clinical data. One of the main challenges is the efficient integration of these data into digital twins, ensuring that all relevant information is available in one place. This can be complicated by the diversity of health record systems and data standards.
• Privacy and Security: Maintaining the privacy and security of health data is a critical concern. Digital twins contain highly sensitive information, and it is essential to ensure that they are protected from unauthorized access and data breaches.
• Interoperability: Healthcare systems often use different technologies and standards. For digital twins to be effective, they need to be interoperable, i.e., able to communicate and share information effectively between different systems.

6. Conclusions

Author contributions, data availability statement, conflicts of interest.

• Ahram, T.; Karwowski, W.; Falcão, C. Applications of Systems Engineering for Testing and Evaluation: A Human Systems Integration Perspective. In APA Handbook of Human Systems Integration ; Boehm-Davis, D.A., Durso, F., Lee, J.D., Eds.; APA handbooks in psychology; American Psychological Association: Washington, DC, USA, 2015; pp. 101–111. [ Google Scholar ] [ CrossRef ]
• Gunal, M.M. (Ed.) Simulation for Industry 4.0 ; Springer: Cham, Switzerland, 2019. [ Google Scholar ] [ CrossRef ]
• Lu, Q.; Xie, X.; Parlikad, A.K.; Schooling, J.M.; Konstantinou, E. Moving from building information models to digital twins for operation and maintenance. Proc. Inst. Civ. Eng. 2021 , 174 , 46–56. [ Google Scholar ] [ CrossRef ]
• Pires, F.; Cachada, A.; Barbosa, J.; Moreira, A.P.; Leitão, P. Digital Twin in Industry 4.0: Technologies, Applications and Challenges. In Proceedings of the 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), Helsinki, Finland, 22–25 July 2019; pp. 721–726. [ Google Scholar ] [ CrossRef ]
• Tao, F.; Zhang, M. Digital twin shop-floor: A New shop-floor paradigm towards smart manufacturing. IEEE Access 2017 , 5 , 20418–20427. [ Google Scholar ] [ CrossRef ]
• Mussomeli, A.; Meeker, B.; Shepley, S.; Schatsky, D. Expecting Digital Twins ; Deloitte Insights: Helsinki, Finland, 2018; Available online: https://www2.deloitte.com/content/dam/insights/us/articles/3773_Expecting-digital-twins/DI_Expecting-digital-twins.pdf (accessed on 6 October 2023).
• Barricelli, B.R.; Casiraghi, E.; Fogli, D. A Survey on Digital Twin: Definitions, Characteristics, Applications, and Design Implications. IEEE Access 2019 , 7 , 167653–167671. [ Google Scholar ] [ CrossRef ]
• Semeraro, C.; Lezoche, M.; Panetto, H.; Dassisti, M. Digital twin paradigm: A systematic literature review. Comput. Ind. 2021 , 130 , 103469. [ Google Scholar ] [ CrossRef ]
• Kritzinger, W.; Karner, M.; Traar, G.; Henjes, J.; Sihn, W. Digital Twin in manufacturing: A categorical literature review and classification. IFAC-PapersOnLine 2018 , 51 , 1016–1022. [ Google Scholar ] [ CrossRef ]
• Rivera, L.F.; Jiménez, M.; Angara, P.; Villegas, N.M.; Tamura, G.; Müller, H.A. Towards continuous monitoring in personalized healthcare through digital twins. In Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering, Toronto, ON, Canada, 4–6 November 2019. [ Google Scholar ]
• Scharff, S. From Digital Twin I’m Improved Patient Experience. Siemens Healthineers. 10 September 2010. Available online: https://www.siemens-healthineers.com/perspectives/mso-digital-twin-mater.html (accessed on 16 October 2023).
• Sun, T.; He, X.; Li, Z. Digital twin in healthcare: Recent updates and challenges. Digit. Health 2023 , 9 , 20552076221149651. [ Google Scholar ] [ CrossRef ] [ PubMed ] [ PubMed Central ]
• Fuller, A.; Fan, Z.; Day, C.; Barlow, C. Digital Twin: Enabling Technologies, Challenges and Open Research. IEEE Access 2020 , 8 , 108952–108971. [ Google Scholar ] [ CrossRef ]
• Armentano, R.; Bhadoria, R.S.; Chatterjee, P.; Deka, G.C. The Internet of Things: Foundation for Smart Cities, eHealth, and Ubiquitous Computing ; Chapman and Hall/CRC: Boca Raton, FL, USA, 2017. [ Google Scholar ]
• Lu, V.Q.; Parlikad, A.K.; Woodall, P. Developing a dynamic digital twin at a building level: Using Cambridge campus as case study. In Proceedings of the 2019 International Conference on Smart Infrastructure and Construction 2019 (ICSIC) Driving Data-Informed Decision-Making, Cambridge, UK, 8–10 July 2019. [ Google Scholar ]
• Grieves, M. Digital twin: Manufacturing excellence through virtual factory replication. In White Paper ; NASA: Greenbelt, MA, USA, 2014. [ Google Scholar ]
• Tsaramirsis, G.; Kantaros, A.; Al-Darraji, I.; Piromalis, D.; Apostolopoulos, C.; Pavlopoulou, A.; Alrammal, M.; Ismail, Z.; Buhari, S.M.; Stojmenovic, M.; et al. A Modern Approach towards an Industry 4.0 Model: From Driving Technologies to Management. J. Sens. 2022 , 2022 , 5023011. [ Google Scholar ] [ CrossRef ]
• Guo, J.; Lv, Z. Application of Digital Twins in multiple fields. Multimed Tools Appl. 2022 , 81 , 26941–26967. [ Google Scholar ] [ CrossRef ] [ PubMed ] [ PubMed Central ]
• Schwartz, S.M.; Wildenhaus, K.; Bucher, A.; Byrd, B. Digital Twins and the Emerging Science of Self: Implications for Digital Health Experience Design and “Small” Data. Front. Comput. Sci. 2020 , 2 , 31. [ Google Scholar ] [ CrossRef ]
• Stojanovic, V.; Trapp, M.; Richter, R.; Hagedorn, B.; Döllner, J.; Alamaniotis, M. Towards the generation of digital twins for facility management based on 3D point clouds. In Proceedings of the 34th Annual ARCOM Conference, Belfast, UK, 3–5 September 2018. [ Google Scholar ]
• Karakra, A.; Fontanili, F.; Lamine, E.; Lamothe, J.; Taweel, A. Pervasive computing integrated discrete event simulation for a hospital digital twin. In Proceedings of the 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA), Aqaba, Jordan, 28 October–1 November 2018; pp. 1–6. [ Google Scholar ]
• Aluvalu, R.; Mudrakola, S.; Maheswari, V.U.; Kaladevi, A.C.; Sandhya, M.V.S.; Bhat, C.R. The novel emergency hospital services for patients using digital twins. Microprocess. Microsyst. 2023 , 98 , 104794. [ Google Scholar ] [ CrossRef ]
• Björnsson, B.; Borrebaeck, C.; Elander, N.; Gasslander, T.; Gawel, D.R.; Gustafsson, M.; Jörnsten, R.; Lee, E.J.; Li, X.; Lilja, S.; et al. Digital twins to personalize medicine. Genome Med. 2019 , 12 , 4. [ Google Scholar ] [ CrossRef ]
• Chaudhari, P.; Gangane, C.; Lahe, A. Digital Twin in Industry 4.0 A Real-Time Virtual Replica of Objects Improves Digital Health Monitoring System. In Information Systems and Management Science, Proceedings of the 4th International Conference on Information Systems and Management Science, Msida, Malta, 14–15 December 2021 ; Springer International Publishing: Cham, Switzerland, 2021. [ Google Scholar ]
• Cheng, W.; Lian, W.; Tian, J. Building the intelligent hospital twins for all-scenario intelligence health care. Digit. Health 2022 , 8 , 20552076221107894. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Safa, E.; Asan, O. Digital Twins for Managing Health Care Systems: Rapid Literature Review. J. Med. Internet Res. 2022 , 24 , e37641. [ Google Scholar ]
• Michelfeit, F. Exploring the Possibilities Offered by Digital Twins in Medical Technology. Siemens Healthcare GmbH-Communc. RSNA. 2018. Available online: https://cdn0.scrvt.com/39b415fb07de4d9656c7b516d8e2d907/1800000005899262/d3fa3e835e2d/Exploring-the-possibilities-offered-by-digital-twins-in-medical-technology_1800000005899262.pdf (accessed on 23 March 2024).
• Gabrielli, S.; Piras, E.M.; Ibarra, O.M. Digital Twins in the Future Design of Digital Therapeutics. In Proceedings of the Adjunct, 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, Cancun, Mexico, 8–12 October 2023. [ Google Scholar ]
• Marshall, T. VPH: The Ultimate Stage Before Your Own Medical Digital Twin. 6 September 2016. Available online: https://www.linkedin.com/pulse/vph-ultimate-stage-before-your-own-medical-digital-twin-marchal/ (accessed on 16 October 2019).
• Vallée, A. Digital twin for healthcare systems. Front. Digit. Health 2023 , 5 , 1253050. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Venkatesh, K.P.; Raza, M.M.; Kvedar, J.C. Health digital twins as tools for precision medicine: Considerations for computation, implementation, and regulation. NPJ Digit. Med. 2022 , 5 , 150. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Viceconti, M.; De Vos, M.; Mellone, S.; Geris, L. Position Paper from the Digital Twins in Healthcare to the Virtual Human Twin: A Moon-Shot Project for Digital Health Research. IEEE J. Biomed. Health Informatics 2023 , 28 , 491–501. [ Google Scholar ] [ CrossRef ]
• World Health Organization. Global Strategy on Digital Health 2020–2025 ; Licence: CC BY-NC-SA 3.0 IGO; World Health Organization: Geneva, Switzerland, 2021. [ Google Scholar ]
• Brazil. Ministry of Health. Executive Secretariat. SUS Information Technology Department. Digital Health Strategy for Brazil 2020–2028 [Electronic Resource]/Ministry of Health, Executive Secretariat, SUS Information Technology Department—Brasília: Ministry of Health, 2020; 128 p.: il. ISBN 978-85-334-2841-6. Available online: http://bvsms.saude.gov.br/bvs/publicacoes/estrategia_saude_digital_Brasil.pdf (accessed on 27 March 2024).
• Garcia, L.P.; Duarte, E. Infodemic: Excess quantity to the detriment of quality of information about COVID-19. Epidemiologia Serv. Saude 2020 , 29 , e2020186. [ Google Scholar ] [ CrossRef ]
• Huang, P.-h.; Kim, K.-H.; Schermer, M. Ethical issues of digital twins for personalized health care service: Preliminary mapping study. J. Med. Internet Res. 2022 , 24 , e33081. [ Google Scholar ] [ CrossRef ]
• Millstone, E.; van Zwanenberg, P. The crisis of trust: For science, scientists or for institutions? Nat. Med. 2000 , 6 , 1307–1308. [ Google Scholar ] [ CrossRef ]
• Barros, M.; Lima, P.; Oliveira, D.; Arcanjo, C.; Oliveira, L.; Pereira, S. Digital Inclusion and Education: Equity and access. Int. J. Sci. Studies 2023 , 1 , 124–149. [ Google Scholar ] [ CrossRef ]
• WHO—World Health Organization. WHO Global Strategy on Integrated People-Centred Health Services 2016-2026—Executive Summary: Placing People and Communities at the Centre of Health Services. [Sl]: Draft for Consultation, Version 2015. Available online: https://iris.who.int/bitstream/handle/10665/180984/WHO_HIS_SDS_2015.20_eng.pdf?sequence=1&isAllowed=y (accessed on 21 September 2023).
• Majumder, S.; Deen, M.J. Smartphone Sensors for Health Monitoring and Diagnosis. Sensors 2019 , 19 , 2164. [ Google Scholar ] [ CrossRef ]
• Ricci, A.; Croatti, A.; Montagna, S. Pervasive and connected digital twins—A vision for digital health. IEEE Internet Comput. 2021 , 26 , 26–32. [ Google Scholar ] [ CrossRef ]
• WHO—World Health Organization. Why Are Health Systems Important? WHR 2000. WHO. Available online: https://www.who.int/publications/i/item/924156198X (accessed on 27 March 2024).
• El-Warrak, L.; Nunes, M.; Luna, G.; Barbosa, C.E.; Lyra, A.; Argôlo, M.; Lima, Y.; Salazar, H.; de Souza, J.M. Towards the future of Public Health: Roadmapping Trends and Scenarios in the Post-COVID Healthcare Era. Healthcare 2023 , 11 , 3118. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• US National Science Foundation. Science and Engineering Indicators 2018 ; National Science Foundation: Alexandria, VA, USA, 2018. Available online: https://www.nsf.gov/statistics/2018/nsb20181/assets/nsb20181.pdf (accessed on 10 July 2024).
• Rodrigues, J.L.d.S.d.Q.; Portela, M.C.; Malik, A.M. Agenda for research on patient-centred care in Brazil. Ciência Saúde Coletiva 2019 , 24 , 4263–4275. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• The Health Foundation. Person-Centred Care Made Simple: What Everyone Should Know about Person-Centred Care. Quick Guide ; The Health Foundation: London, UK, 2016; Available online: https://www.health.org.uk/publications/person-centred-care-made-simple (accessed on 20 September 2023).
• Wickramasinghe, N.; Jayaraman, P.P.; Forkan, A.R.M.; Ulapane, N.; Kaul, R.; Vaughan, S.; Zelcer, J. A Vision for Leveraging the Concept of Digital Twins I’m Support the Provision of Personalized Cancer Care. IEEE Internet Comput. 2022 , 26 , 17–24. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

El-Warrak, L.; de Farias, C.M. The State of the Art of Digital Twins in Health—A Quick Review of the Literature. Computers 2024 , 13 , 228. https://doi.org/10.3390/computers13090228

El-Warrak L, de Farias CM. The State of the Art of Digital Twins in Health—A Quick Review of the Literature. Computers . 2024; 13(9):228. https://doi.org/10.3390/computers13090228

El-Warrak, Leonardo, and Claudio M. de Farias. 2024. "The State of the Art of Digital Twins in Health—A Quick Review of the Literature" Computers 13, no. 9: 228. https://doi.org/10.3390/computers13090228

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Open access
• Published: 11 September 2024

Incorporating equity, diversity and inclusion (EDI) into the education and assessment of professionalism for healthcare professionals and trainees: a scoping review

• Darsh Shah 1 ,
• Nima Behravan 1 ,
• Nujud Al-Jabouri 2 &
• Matthew Sibbald 1 , 2  

BMC Medical Education volume  24 , Article number:  991 ( 2024 ) Cite this article

Metrics details

Current definitions of professionalism for healthcare trainees often lack equity, diversity and inclusion (EDI) in the expectations and assessment of professionalism. While professionalism teaching is incorporated in healthcare training, equity-deserving groups still experience discrimination. This scoping review investigates the literature to understand how EDI and associated domains of cultural humility, and advocacy can be incorporated in healthcare trainees’ education and assessment of professionalism.

The Arksey and O’Malley framework was applied to this scoping review. MEDLINE, Embase & PsychINFO were searched up to March 2023, with terms surrounding health professionals, professionalism, EDI, cultural humility, and advocacy. Titles and abstracts ( n  = 3870) and full-texts ( n  = 140) were independently screened by two reviewers. Articles were included if they focused on EDI, cultural humility, or advocacy among healthcare students/trainees, and had outcomes related to professionalism. Articles lacking discussion of professionalism as an outcome were excluded. Themes were generated by mutual discussion. Risk of bias was assessed using the Cote et al. and Medical Education Research Study Quality Instrument (MERSQI) tools.

48 articles underwent thematic analysis. Studies investigated the disciplines of medicine, nursing, social work, physiotherapy, and dentistry. Most studies were qualitative in methodology ( n  = 23). Three themes emerged: (1) EDI-related interventions are associated with improved professionalism of healthcare trainees/workers ( n  = 21). Interventions employed were either an EDI-associated educational course ( n  = 8) or an exchange program to promote EDI competencies among trainees ( n  = 13). (2) Trainee definitions and perceptions of professionalism include themes related to EDI and cultural humility ( n  = 12). (3) Current standards of professionalism are perceived as non-inclusive towards historically-marginalized populations ( n  = 15). Literature investigating advocacy as it relates to professionalism is limited.

This review identified that core EDI principles and its associated domains of cultural humility and advocacy are often viewed as integral to professionalism. These findings create a strong impetus to incorporate EDI principles within professionalism frameworks in healthcare education. Future research should employ standardized tools for professionalism assessment to provide more conclusive evidence. Incorporating patient perspectives of professionalism can inform actionable recommendations for fostering inclusive healthcare environments.

Peer Review reports

Introduction

In the healthcare pedagogical literature, professionalism has often been broadly defined as a set of characteristics, competencies and attitudes that are expected of a healthcare professional or trainee [ 1 ]. A paper published in the American Medical Association Journal of Ethics suggests that upholding equity is amongst the most important roles of a physician by stating: “Organizational, system, and policy reform demand that professionalism be redefined in terms of its capacity to motivate equity in health professions education and clinical practice” [ 2 ]. However, despite attempts of healthcare institutions to define specific domains and competencies in their professionalism frameworks, concepts of EDI, cultural humility and advocacy are often omitted in these frameworks [ 2 ].

An example of a healthcare professionalism framework is one developed by the Michael G. DeGroote School of Medicine (MGDSM) at McMaster University in Hamilton, Ontario, Canada. This framework is called the Professionalism in Practice (PIP) framework, which teaches and holds its learners accountable to the following core domains of professionalism: professional responsibility and integrity, pursuit of excellence/insight, personal interactions, as well as EDI and Indigenous reconciliation [ 3 ]. As noted here, the MGDSM makes a deliberate effort to outline EDI and Indigenous reconciliation as a core domain to uphold as a standard of its learners’ and providers’ professionalism. Another example of a healthcare professionalism framework is that developed by the National Taiwan University College of Medicine (NTUCM). The NTUCM developed this medical professionalism framework through contributions from stakeholders, including chairs of hospital departments, residents, and attending physicians. The resultant consensus framework consisted of 8 domains: integrity, humanism, altruism, communication, clinical competence, ethics, excellence, and accountability [ 4 ]. Despite some overlap between the healthcare professionalism frameworks of NTUCM and MGDSM, various competencies are not shared across the two frameworks. For example, the NTUCM framework does not explicitly have a domain pertinent to EDI, cultural humility or advocacy [ 4 ]. Therefore, due to the inherent subjectivity and lack of unison in the definitions and domains of professionalism frameworks at different institutions, crucial concepts such as EDI, cultural humility and advocacy may be left out of professionalism curricula and frameworks.

Numerous studies find that EDI, cultural humility and advocacy are beneficial for both patients and healthcare teams. A review paper found that multiple studies suggest patient outcomes are improved by more diverse teams, and that healthcare environments that are identified as diverse are found to be less prone to disputes in times of change [ 5 ]. Additionally, despite the efforts of healthcare institutions in teaching, assessing and upholding professionalism expectations from their trainees, discriminatory practices and consequent healthcare disparities still persist. For instance, in Canada, Black and Indigenous communities continue to experience discrimination when seeking healthcare services [ 6 , 7 , 8 , 9 , 10 ]. In an ethnographic study conducted in an emergency department (ED) in a large teaching hospital in a Western Canadian city, it was found that many Indigenous patients felt they were being judged on the basis of their identity, and that presumptions were made that their visits to the ED were due to illegitimate pain issues or inappropriate reasons [ 6 ]. An important example of discriminatory healthcare practices against Indigenous communities is that of Joyce Echaquan, an Indigenous patient at a Quebec hospital who, moments before her passing, recorded a video displaying her screaming in pain while her healthcare providers made distasteful remarks towards her based on racially-charged stereotypes [ 7 ]. Additionally, a qualitative study on self-identified Black individuals who lived in Montréal during the COVID-19 pandemic found several themes regarding internalized anti-Black racism amongst healthcare providers, including the presence of insensitivity towards racial discrimination by some providers [ 8 ].

The discriminatory practices and healthcare disparities amongst patients of equity-deserving backgrounds noted above [ 6 , 7 , 8 , 9 , 10 ], combined with the lack of emphasis on EDI in healthcare professionalism definitions [ 2 ], creates a strong impetus for redefining expectations of professionalism from healthcare professionals and trainees to encompass concepts of cultural humility and advocacy.

This scoping review systematically searches the literature to evaluate how EDI and associated domains of cultural humility and advocacy are related to and can be incorporated into the standards, teaching and evaluation of professionalism for healthcare workers and trainees.

Research question

This scoping review aggregates the existing evidence on the following research question: What literature is available on how principles of equity , diversity and inclusion (EDI) are incorporated in the education and assessment of professionalism for health professions trainees?

Approach to terminology

This paper utilizes the terminology recommended by the American Medical Association (AMA) and the Association of American Medical Colleges (AAMC) to guide language, narrative, and concepts as our standard for inclusive language that promotes health equity. For instance, per the AMA and AAMC recommendations, the terms “cultural competency”, “minority/minority groups” and “equality” as commonly found in the literature, are substituted with terms such as “cultural humility/safety”, “(people from) racial and ethnic groups’’ and “equity”, respectively [ 11 ].

Materials and methods

We adopted scoping review methodology to understand the breadth and depth of literature pertaining to the principles of EDI in professionalism education and assessment [ 12 ]. We were guided by Arksey and O’Malley’s methodological framework for scoping reviews, which includes five main stages (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; and (5) collating, summarizing and reporting the results [ 13 ]. We reported our process according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist through multiple rounds of feedback and modification by our research team [ 14 ].

Eligibility criteria

Studies were included if (1) the population was practicing health professionals and trainees working in the fields of medicine, nursing, social work, rehabilitation, multidisciplinary healthcare teams, medical education, dentistry, midwifery, and pharmacy (2) the outcome was related to professionalism within the objectives, methodology, or results sections, and (3) professionalism was linked to EDI, cultural humility, or advocacy. Studies were excluded if they (1) were not in English, (2) did not focus on professionalism outcomes, (3) did not focus on our population of interest, and (4) were grey literature publications (e.g. conference proceedings, abstracts, non-peer reviewed reports). Considering that both empirical and non-empirical studies have the potential to yield meaningful findings on concepts of EDI and professionalism, no restrictions were placed on study design.

Search strategy

A preliminary search was conducted to identify keywords and subject headings related to our research question. The final search strategy was developed in consultation among the research team members and a librarian at McMaster Health Sciences Library. Three electronic databases were searched from inception to March 7, 2023: Embase, Medline and PsycINFO. Search terms were designed to capture concepts of “advocacy” AND “cultural humility” AND “healthcare professionals” AND “healthcare trainees” AND “professionalism” AND “professional development.” Search terms were adapted for each database as subject headings or keywords where appropriate (See Appendix S1 , Additional File 1 ). To identify any non-indexed literature, we hand searched the reference list of included studies and Google Scholar. The search strategy used the same search term combinations as described above and was limited to literature published in peer-reviewed journals only. Grey literature, such as individual institution’s professionalism frameworks, were not included. Covidence software was used to manage citations, including removing duplicates and screening [ 15 ].

A revised search was conducted between March 7th 2023 to April 1st 2024 with an identical search strategy to identify advances in the literature. A set of 204 unique records were identified for which abstract and full-text screening were completed. We did not identify any new studies meeting inclusion/exclusion criteria that added substantial evidence to the results of our study.

Screening process

A pilot screening test of the eligibility criteria was conducted by two independent reviewers (NA, DS, or NB) on a small sample of studies. Reviewers met to discuss their agreement level and the eligibility criteria was modified for clarity. Following this pilot test, two independent reviewers (NA, DS, or NB) screened titles and abstracts to evaluate their eligibility against the inclusion and exclusion criteria. Studies that passed the initial screening process underwent full-text screening by two independent reviewers (NA, DS, or NB). Disagreements at any screening stage were resolved by a third reviewer or consensus-based discussion. A PRISMA-ScR flowchart was used to show the process of study selection (Fig.  1 ).

Data extraction

A data extraction sheet was developed by the authors and used to organize data from included studies. Data extraction was divided among three members (NA, DS or NB) and was conducted using the Excel software. All extracted data was reviewed by a different member (NA, DS or NB). The final extraction table included (1) study characteristics (i.e., authors, publication year, country), (2) sample characteristics (i.e., number of participants, population description), (3) methodological characteristics (i.e., study design, program description/recruitment), and (4) study outcomes related to professionalism (See Appendix S2 , Additional File 1 ).

Quality appraisal

One reviewer (NA, DS or NB) assessed the quality of studies using tools for qualitative and quantitative data. This consisted of the Medical Education Research Study Quality Instrument (MERSQI) for quantitative studies, and a qualitative study grid published by Côté et al. [ 16 , 17 ].

Data analysis

This review qualitatively analyzed the data following the thematic analysis approach [ 18 ]. Three reviewers (NA, DS & NB) independently categorized data according to their meaning and content. Subsequently, key themes were independently identified and formulated. These themes were then compared and any disagreements were resolved with discussion-based consensus, arriving at three main themes.

Literature scope and characteristics

Results of the initial search are presented in Fig.  1 , including rationales for excluded articles. The data searches yielded 4194 unique records, with 125 selected for full-text review. After screening against eligibility criteria, 34 articles were included for data extraction. Included articles spanned publication between 2005 and 2023. An additional 24 articles were identified from hand-searching of included studies, of which 14 met eligibility criteria, producing a total of 48 articles included in our analysis.

PRISMA-ScR Flow Diagram

Most studies involved medical trainees or physicians ( n  = 18). Other professions examined in the literature included nursing, social work, dentistry, midwifery, and physical therapists. Most studies were based in the United States ( n  = 19) among other countries such as Australia ( n  = 4), the United Kingdom ( n  = 1), Canada ( n  = 4), and Turkey ( n  = 3) with eleven studies being multinational ( n  = 11). The majority of studies were qualitative in nature ( n  = 23) followed by mixed-methods ( n  = 10), commentaries ( n  = 8), and quantitative ( n  = 7). Characteristics of the studies divided by themes can be found in the supplementary material (Appendix S2 , Additional File 1 ) Among qualitative studies, methods of evaluating professionalism were predominantly based on participant interviews ( n  = 16) followed by surveys ( n  = 13), written narratives ( n  = 9), and literature review ( n  = 3). Quantitative measures included internally designed Likert scales to measure participants’ perspectives ( n  = 14) with few studies employing validated tools ( n  = 3). Twenty-one studies employed an EDI training intervention, split between educational courses ( n  = 8) and clinical placements ( n  = 11). A summary of quality appraisal is presented in the supplementary material (See Appendix S3 , Additional File 1 ).

Theme 1: EDI-related interventions are associated with improved professionalism of healthcare trainees/professionals

Most studies investigated outcomes after an EDI-oriented intervention on professionalism outcomes among participants ( n  = 21). Studies within this theme are divided based on their intervention of an educational course ( n  = 8) or an exchange clinical placement ( n  = 13). Topics of the educational courses included health advocacy, cultural humility, and spirituality/religion [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ]. All eight studies reported an improvement in professionalism or professional identity formation. Four of the eight studies explicitly link professionalism and EDI concepts [ 19 , 20 , 22 , 26 ]. In one study, first-year medical students completed a 10-month community health elective course aimed to expand cultural humility and advocacy in the context of adolescent care [ 22 ]. Reflective essays completed at the end of the course revealed that 90% of students noted an increase in knowledge and skills of professionalism [ 22 ]. These findings are seen across healthcare disciplines including nursing, pharmacy, social work, and other allied health. A 10-week interprofessional course aimed at developing cultural humility induced professional growth in addition to improved cultural proficiency among nursing, pharmacy, and social work trainees [ 26 ]. These studies demonstrate improved professionalism as an outcome for EDI-promotion through healthcare education.

Studies employing clinical placement interventions as methods of EDI-training revolved around either rural community or international placements [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 ]. Professionalism outcomes included a heterogenous combination of reflections on patient-clinican and interprofessional interactions [ 32 , 34 ]. All thirteen studies report improvement in domains of professionalism secondary to exchange programs within different cultural settings [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 ]. A 4-week exchange between Japanese and UK medical students was associated with professionalism outcomes including social justice and resource stewardship [ 38 ]. Five of the thirteen studies explicitly connect domains of EDI to professionalism or professional development [ 28 , 29 , 33 , 36 , 37 ]. In these cases, cross-cultural experiences directly improve professionalism in participants [ 31 , 33 , 36 ]. Remaining studies describe parallel improvements in cultural humility and professionalism as outcomes of the intervention with implicit associations between the two concepts [ 27 , 39 ].

Theme 2: operationalization of professionalism revealed themes of EDI

A second subset of studies assessed trainee conceptualization of health professionalism within their respective field of practice ( n  = 12) [ 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 ]. These studies investigate the attitudes, perspectives, and competencies of trainees on professionalism in relationship to domains of EDI. Trainees note that adaptibility and humility are critical elements of professionalism, which extends to adapting to cultural and social norms [ 51 ]. Inability to accomodate differences in gender norms, language, or cultural beliefs are sources of professionalism dilemmas [ 44 ]. Hamdan Alshammari and Alboliteeh applied a structural equation model to questionnaire responses of 587 nurses in Saudi Arabia. They find significant correlations between dimensions of professionalism and cultural competency [ 50 ]. Similarly, practitioner perceptions of cultural competency are closely related to perceptions of professional development rather than formal health professional education [ 43 ]. A bidirectional relationship between professionalism and themes of EDI is demonstrated in the literature. Topics of EDI including cultural humility and health advocacy are often cited as core components of professional development [ 47 , 51 ]. Simultaneously, medical trainees, physicians, and researchers identify a need to integrate professionalism assessment within the development of an effective cultural humility curricula [ 46 ]. Conventional professionalism values such patient centeredness, bias recognition and clinical skills are necessary for cultural humility [ 46 ]. Therefore, not only is EDI training a contributing factor to professionalism, professional identity development is required for acquisition of EDI competencies.

Theme 3: current standards of professionalism are perceived as non-inclusive

The final set of studies explored the perspectives of trainees and practitioners from equity-deserving groups (n= [ 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 ]. Specifically, researchers investigated perceptions of professionalism in the workplace with reference to gender identity, sexual orientation, race/ethnicity and other EDI-associated demographic factors [ 52 , 53 ]. Participants from equity-deserving groups routinely experience professionalism as culturally and ethnically restrictive [ 53 ]. In these studies, professionalism is defined as a reflection of the cultural norms and expectations of the dominant social group [ 59 ]. Given the historical context in which professionalism was established, these norms often stem from the “White male identity” [ 53 ]. Consequently, trainees and practitioners that deviate from existing definitions of professionalism in terms of racial or ethnic identity, culture, skin colour, gender identity, sexual orientation, or colour are at a disadvantage. Survey of practitioners from equity-deserving backgrounds notes increased criticism over professional actions and increased pressure to conform. Furthermore, cultural and ethnic incongruity with the professionalism standard inhibits professional development to positions of leadership [ 54 , 55 ]. This literature is also surrounded by a handful of recent commentaries by physicians on the current inequities faced by equity-deserving populations [ 59 , 60 , 61 , 62 , 63 , 64 , 65 ]. Rosenberg and colleagues discuss three cases in which professionalism standards propagated disparities within medicine. One case is of a Latin medical student who was deemed unprofessional in a clinical examination as she was wearing Latin earrings [ 61 ]. Dr. AbdelHameid shares her experiences as a Black physician burden by an expectation to comply with racially charged interactions with patients and colleagues in order to align with professionalism expectations [ 62 ]. International medical graduates highlight the unique challenges of adapting to professionalism norms due to the inherent interconnectedness of professional identity and cultural schemata [ 65 ]. A common motif remains a sense of rigidity in definitions of professionalism with limited ethnic expression afforded to practitioners [ 59 , 62 ]. Accordingly, there is increasing support for the existing definitions of professionalism to include measures of equity, diversity, and cultural humility [ 52 , 55 , 56 ]. Professionalism is viewed as a tool to promote anti-discriminatory practice within the clinical landscape [ 54 ]. Advocacy is proposed as a valuable competency for trainees to induce changes in institutional professionalism standards [ 53 ]. Strategies to centre equity and inclusion include increased justice within professionalism assessment and greater value for diverse populations on clinical teams [ 65 ]. Patient facing strategies outlined by studies within this theme include greater humility for structural inequities faced by marginalized populations and attention to positionality in patient care (Fig.  2 ) [ 65 ].

The aim of this scoping review was to characterize existing literature surrounding EDI, cultural humility, and advocacy principles in professionalism education, assessment, or experiences for healthcare trainees and practitioners. EDI and professionalism are explored in three predominant modalities. EDI-interventions such as educational courses improve professionalism outcomes among learners (Fig.  2 ) [ 19 , 22 , 24 ]. In the present studies, rural and foreign exchange programs are intended to produce improvements in cultural humility. Previous research supports the beneficial outcomes of short-term overseas programs in cultural humility [ 65 , 67 ]. The present studies extend this finding by demonstrating concurrent improvements in professionalism [ 32 , 34 , 38 ]. Furthermore, there is a direct and positive correlation between professional and cultural competencies measured among practitioners [ 50 ]. These findings further support an interconnected model of professionalism and cultural humility.

Integrated model of professionalism including EDI, cultural humility, and health advocacy

Interventions promoting advocacy as a domain of professionalism were more limited in the literature [ 20 ]. While there is a strong agreement for advocacy as a professional responsibility, the scope of the healthcare practitioner as an advocate remains unclear and may contribute to the lack of organized interventions in this area [ 68 , 69 , 70 , 71 ]. Nevertheless, Peluso and colleagues (2013) discuss the success of a four-week “advocacy and activism” module as a component of medical students’ professionalism education. Advocacy is frequently cited within the scope of professional competencies of healthcare trainees [ 72 , 73 ]. Therefore, structured assessment of advocacy education in professionalism remains an area requiring further investigation.

Development and revision of professionalism frameworks rely on values expressed by trainees and practitioners [ 73 , 74 ]. Studies in the first and second domains of our findings cumulatively express an inherent relationship between EDI and professionalism. Interventions to promote EDI competencies, whether it be through educational courses or clinical placements, foster professionalism amongst learners. Healthcare trainees and professionals conceptualize EDI competencies as core values of professionalism in contrast to existing frameworks that lack EDI domains [ 45 , 75 ]. Conversely, healthcare practitioners with formed professional identities tend to demonstrate greater awareness for EDI [ 39 ]. An application of social learning theory translated from engineering profession consolidates this bidirectional relationship in which successful professional identity formation is necessary to acquire values of EDI [ 76 ]. These findings suggest that EDI and professionalism are interrelated domains rather than mutually exclusive competencies.

Commentaries around this topic in the literature signify the disparity faced by individuals from equity-deserving groups due to the lack of EDI integration in current professionalism standards. Narratives from equity deserving groups recall experiences of suppressing their cultural, racial, sexual, and other personal identities to conform with definitions of professionalism. Expression of personal identity that doesn’t align with definitions of professionalism is viewed as unprofessional, hinding individuals from equity deserving groups from attaining positions of leadership [ 62 ]. The coin model of privilege and critical allyship, as described by Nixon, conceptualizes each system of inequity as a coin. These coins provide unearned power to certain individuals based on their relationship to the system of inequity. It elegantly highlights the disparity in power to reform systems of inequity disproportionately given to populations of historic domination [ 77 ]. In our context, the coin signifies existing standards of professionalism by which health professionals are trained and held accountable. Current standards are viewed as exclusive to professionals from equity-deserving groups, rendering their placement on their bottom of the professionalism “coin” and resulting in an unearned disadvantage. The path towards critical allyship requires efforts from those in positions of privilege to reshape these frameworks, incorporating greater awareness of domains of EDI as core competencies of the health professional. Greater emphasis on the values of EDI in the professionalism standards will help bridge health disparities, reflected in improved patient outcomes and higher quality care [ 78 , 79 ]. Current challenges to enacting these changes stems from a performative attitude towards professionalism. Professionalism, if reduced to a checklist of behaviours, fails to capture the commitment to social justice required to institute change [ 63 ]. Attitudes and beliefs of professionalism assessment also remains a challenge, as trainees are more hesitant to engage in advocacy if their institutions are viewed as hostile towards advocacy efforts [ 53 ]. Engaging trainees in the design of professionalism education may prove to be instrumental in instigating change [ 53 ].

Limitations of the literature

Most studies assessed changes in professionalism or measures of EDI with self-reported techniques such as questionnaires, written reflections, or interviews. A few studies use validated tools for assessment of professional development [ 22 ] while most rely on the interpretation of student feedback or internally constructed questionnaires [ 25 , 26 ]. Over the last three decades, an increasing number of validated professionalism tools have been developed based on existing frameworks of professionalism [ 79 , 80 ]. Professionalism assessment inventories are available across medicine, nursing, and pharmacy and demonstrate high reliability and content validity compared to self-report measures [ 80 , 81 , 82 ]. However these tools remain scarce, in-part due to unclear definitions of professionalism, and thus self-report measures are favoured [ 83 ]. Increased use of these tools would support more valid assessments of professionalism in the context of EDI-associated interventions. Studies present in this review engaged solely health trainees and professionals with self-assessment of professionalism. Perspectives from patients are a valuable, and often underutilized, source for gauging professionalism in clinical settings [ 75 ]. Studies investigating patient perception of professionalism in the context of EDI training may serve as a valuable measure for the integration of these concepts.

Limitations of our study

The literature search was developed according to the starting framework of the professionalism-in-practice (PIP) developed at McMaster University [ 3 ]. The framework provides guidance towards query terms related to domains of EDI. Therefore, the conceptualization of the present literature is within the context of the applied framework. Other frameworks may have informed a different strategy to investigate literature on professionalism [ 73 , 84 ]. Secondly, our study aims to promote more equity-oriented language according to the AAMC guide to language promoting health equity [ 11 ]. In this context, we opted to use terms such as “cultural humility” in place of “cultural competency”. However, we acknowledge that those terms are not interchangeable in all contexts [ 85 ].

Conclusion and future directions

To our knowledge, our study is the first to review literature around EDI and professionalism in the healthcare setting. We demonstrate that a significant body of research supports the integration of domains of EDI into professionalism education and assessment across interdisciplinary programs. Interventions aimed to improve measures of EDI concordantly improved measures of professionalism. Domains of EDI, cultural humility, and advocacy correlate with professionalism noted both by self-reported measures and quantitative surveys of health professionals. Current models of professionalism are viewed as non-inclusive to practitioners of equity-deserving groups. These juxtaposing findings suggest an increasing need for the revision of definitions of professionalism to better address competencies of EDI across healthcare disciplines.

We expect this study to drive future research and serve as a support for the development and revision of professionalism frameworks with domains of EDI, cultural humility, and advocacy. The Michael G. DeGroote school of Medicine from which our work is based recently revised its internal model of professionalism to include a domain of EDI [ 3 ]. This revision aims to further address the disparities faced by patients and practitioners of equity-deserving backgrounds. We recommend revisions in professionalism frameworks to serve as a foundation for deeper implementation of EDI in existing health professions training through various modalities including educational opportunities, tools, and mentorship programs. The literature reviewed in this study support the use of interprofessional educational courses. Alternatively electives that increase exposure to equity deserving groups is a feasible form of EDI training for professionalism development. EDI is also increasingly being incorporated into healthcare simulation with recent development of tools for trainees to reflect on simulation training from an EDI perspective [ 85 , 86 , 87 ]. In addition to educational courses and clinical placements discussed in this review, individualized support towards professional identity formation of trainees from equity deserving groups is in emergently recognized aspect of professionalism education [ 88 ]. The University of Toronto’s diversity mentorship initiative have successfully aimed to support professionalism among students from equity-deserving groups, demonstrating the effectiveness of structured mentorship in fostering professional growth within these populations [ 88 ]. Incorporation of EDI into professionalism education for health professionals is a longitudinal endeavour starting at revision of existing frameworks and definitions of professionalism., We hope our work drives the evidence-based design of professionalism frameworks guiding health professional education and standards of assessment.

Data availability

The authors confirm that all relevant data, including the search strategy, databases used, and the papers used in data extraction and analysis are included in the article, or can be found in the supplementary files.

Abbreviations

Equity, Diversity and Inclusion

Medical Education Research Study Quality Instrument

Michael G. DeGroote School of Medicine

Nominal Group Technique

National Taiwan University College of Medicine

Professionalism in Practice

Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews

Lashkari M, Beigzadeh A. The concept and challenges of medical professionalism. Rep Health Care. 2014;1(1):43–5.

Google Scholar  

Chen C, Anderson A. How should Health Professionalism be redefined to address Health Equity? Am Med Association J Ethics. 2021;23(3):E265–270. https://doi.org/10.1001/amajethics.2021.265 .

Article   Google Scholar  

Professionalism in Practice Michael G. DeGroote School of Medicine. Available online: https://pgme.mcmaster.ca/app/uploads/2022/12/Professionalism-in-Practice-Revised-Dec-2022.pdf . Accessed 7 Jan 2024.

Ho MJ, Yu KH, Hirsh D, Huang TS, Yang PC. Does one size fit all? Building a framework for medical professionalism. Acad Med. 2011;86(11):1407–14. https://doi.org/10.1097/ACM.0b013e31823059d1 .

Gomez LE, Bernet P. Diversity improves performance and outcomes. J Natl Med Assoc. 2019;111(4):383–92. https://doi.org/10.1016/j.jnma.2019.01.006 .

Browne AJ, Smye VL, Rodney P, Tang SY, Mussell B, O’Neil J. Access to Primary Care from the perspective of Aboriginal patients at an Urban Emergency Department. Qual Health Res. 2011;21(3):333–48. https://doi.org/10.1177/1049732310385824 .

Shaheen-Hussain S, Lombard A, Basile S. Confronting Medical Colonialism and Obstetric Violence in Canada. Lancet. 2023;401(10390):1763–5. https://doi.org/10.1016/S0140-6736(23)01007-3 .

Williams KKA, Baidoobonso S, Haggerty J, Lofters A, Adams AM. Anti-black discrimination in Primary Health Care: a qualitative study exploring internalized racism in a Canadian context. Ethn Health. 2024;29(3):343–52. https://doi.org/10.1080/13557858.2024.2311429 .

Boyer Y. Healing Racism in Canadian Health Care. Can Med Assoc J. 2017;189(46):E1408–9. https://doi.org/10.1503/cmaj.171234 .

Browne AJ, Fiske JA. First Nations women’s encounters with Mainstream Health Care services. West J Nurs Res. 2001;23(2):126–47. https://doi.org/10.1177/019394590102300203 .

American Medical Association and Association of American Medical Colleges. Advancing Health Equity: Guide on Language, Narrative and Concepts. ama-assn.org/equity-guide . Accessed 7 Jan 2024.

Mak S, Thomas A. Steps for conducting a scoping review. J Grad Med Educ. 2022;14:565–7. https://doi.org/10.4300/JGME-D-22-00621.1 .

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. J Soc Res Methodol. 2005;8:19–32. https://doi.org/10.1080/1364557032000119616 .

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169:467–73. https://doi.org/10.7326/M18-0850 .

Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. 2024. www.covidence.org . Accessed 16 Mar 2023.

Cook DA, Reed DA. Appraising the quality of medical education research methods: the Medical Education Research Study Quality Instrument and the Newcastle-Ottawa Scale-Education. Acad Med. 2015;90:1067–76. https://doi.org/10.1097/ACM.0000000000000786 .

Côté L, Turgeon J. Appraising qualitative research articles in medicine and medical education. Med Teach. 2005;27:71–5. https://doi.org/10.1080/01421590400016308 .

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:1–0. https://doi.org/10.1186/1471-2288-8-45 .

Midik O, Basar F, Emirzeoglu M, Emirzeoglu M, ALAYLI G, Alayli G, Sarisoy G, SARISOY G, Gunbey E, GUNBEY E, Yuce M. An elective program in medical education: communication with Hard of hearing people and Turkish sign Language. Marmara Med J. 2016;29(1):14–22. https://doi.org/10.5472/MMJoa.2901.03 .

Midik O, Coskun O. Medical students’ conceptualisation of Health Advocacy in an interactive Programme. Interact Learn Environ. 2020;28:729–43. https://doi.org/10.1080/10494820.2018.1552876 .

Kaçan CY. Ö. Örsal 2020 Effects of Transcultural Nursing Education on the Professional values, empathic skills, Cultural Sensitivity and Intelligence of students. J Commun Health Nurs 37 65–76 https://doi.org/10.1080/07370016.2020.1736374 .

Lee MS, Kaulukukui SK, Smith MS, Laimana JJ, Voloch KA. Professional and cultural development of medical students mentoring adolescents in a predominately native hawaiian community. Hawai’i J Health Social Welf. 2019;78(12 Suppl 3):35.

Fleming T, Creedy DK, West R. The influence of yarning circles: a Cultural Safety Professional Development Program for midwives. Women Birth. 2020;33:175–85. https://doi.org/10.1016/j.wombi.2019.03.016 .

Rosen D, McCall J, Goodkind S. Teaching critical self-reflection through the Lens of Cultural Humility: an assignment in a Social Work Diversity Course. Social Work Educ. 2017;36:289–98. https://doi.org/10.1080/02615479.2017.1287260 .

Awaad R, Ali S, Salvador M, Bandstra BA. Process-oriented Approach to Teaching Religion and Spirituality in Psychiatry Residency Training. Acad Psychiatry. 2015;39:654–60. https://doi.org/10.1007/s40596-014-0256-y .

Brown B, Warren NS, Brehm B, Breen P, Bierschbach JL, Smith R, Wall A, Loon RAV. The design and evaluation of an Interprofessional Elective Course with a Cultural competence component. J Allied Health 2008, 37.

Daly M, Perkins D, Kumar K, Roberts C, Moore M. What factors in Rural and Remote Extended Clinical Placements May contribute to preparedness for practice from the perspective of students and clinicians? Med Teach. 2013;35:900–7. https://doi.org/10.3109/0142159X.2013.820274 .

Glaser J, Pfeffinger A, Quan J, Fernandez A. Medical students’ perceptions of and responses to Health Care disparities during clinical clerkships. Acad Med. 2019;94:1190–6. https://doi.org/10.1097/ACM.0000000000002582 .

Jansen MB, Lund DW, Baume K, Lillyman S, Rooney K, Nielsen DS. International Clinical Placement – experiences of nursing students’ Cultural, Personal and Professional Development; a qualitative study. Nurse Educ Pract. 2021;51:102987. https://doi.org/10.1016/j.nepr.2021.102987 .

Knecht LD, Wilson KJ, Linton ME, Koonmen JM, Johns EF. Assessing student expectations and perceptions of a short-term International Service‐learning experience. Public Health Nurs. 2020;37:121–9. https://doi.org/10.1111/phn.12669 .

Yao CA, Swanson J, McCullough M, Taro TB, Gutierrez R, Bradshaw A, Campbell A, Magee WP, Magee WP. The Medical Mission and Modern Core Competency Training: a 10-Year Follow-Up of Resident experiences in global plastic surgery. Plast Reconstr Surg. 2016;138:e531–8. https://doi.org/10.1097/PRS.0000000000002484 .

Żebryk P, Przymuszała P, Nowak JK, Cerbin-Koczorowska M, Marciniak R, Cameron H. The impact of ERASMUS exchanges on the Professional and Personal Development of Medical Students. IJERPH. 2021;18:13312. https://doi.org/10.3390/ijerph182413312 .

Koskinen L, Campbell B, Aarts C, Chassé F, Hemingway A, Juhansoo T, Mitchell MP, Marquis FL, Critchley KA, Nordstrom PM. Enhancing Cultural competence: Trans-Atlantic experiences of European and Canadian nursing students. Int J Nurs Pract. 2009;15:502–9. https://doi.org/10.1111/j.1440-172X.2009.01776.x .

Behar-Horenstein LS, Feng X, Roberts KW, Gibbs M, Catalanotto FA, Hudson-Vassell CM. Developing Dental Students’ awareness of Health Care disparities and Desire to serve vulnerable populations through Service‐Learning. J Dent Educ. 2015;79:1189–200. https://doi.org/10.1002/j.0022-0337.2015.79.10.tb06012.x .

Green BF, Johansson I, Rosser M, Tengnah C, Segrott J. Studying abroad: a multiple case study of nursing students’ International experiences. Nurse Educ Today. 2008;28:981–92. https://doi.org/10.1016/j.nedt.2008.06.003 .

Hande DN, Sångberg F, Ulvedal K, Kindblom K. The Experience of Exchange Studies in Physical Therapy between Sweden and India – A Qualitative Study. 2022; https://doi.org/10.36848/PMR/2022/50100.51035

Lee RLT, Pang SMC, Wong TKS, Chan MF. Evaluation of an Innovative Nursing Exchange Programme: Health Counselling skills and Cultural Awareness. Nurse Educ Today. 2007;27:868–77. https://doi.org/10.1016/j.nedt.2006.12.002 .

Nishigori H, Otani T, Plint S, Uchino M, Ban NI, Came I, Saw I, Reflected. A qualitative study into learning outcomes of International Electives for Japanese and British Medical Students. Med Teach. 2009;31:e196–201. https://doi.org/10.1080/01421590802516764 .

Clark Callister L, Harmer Cox A. Opening our hearts and minds: the meaning of international clinical nursing Electives in the Personal and Professional lives of nurses. Nurs Health Sci. 2006;8:95–102. https://doi.org/10.1111/j.1442-2018.2006.00259.x .

Chae D, Park Y, Kang K, Kim JA. Multilevel Investigation of Cultural competence among South Korean clinical nurses. Scandinavian Caring Sci. 2020;34:613–21. https://doi.org/10.1111/scs.12760 .

Adams J, Lui C-W, Sibbritt D, Broom A, Wardle J, Homer C. Attitudes and Referral Practices of Maternity Care Professionals with Regard to complementary and alternative medicine: an integrative review: attitudes and referral of Maternity Care professionals to CAM. J Adv Nurs. 2011;67:472–83. https://doi.org/10.1111/j.1365-2648.2010.05510.x .

Fleming T, Creedy DK, West R. Cultural Safety Continuing Professional Development for Midwifery academics: an Integrative Literature Review. Women Birth. 2019;32:318–26. https://doi.org/10.1016/j.wombi.2018.10.001 .

Teasley ML. Perceived levels of cultural competence through Socail Work Education and Professional Development for Urban School Social Workers. J Social Work Educ 2005, 41, 85–98; https://doi.org/10.5175/JSWE.2005.200300351

Ho M-J, Gosselin K, Chandratilake M, Monrouxe LV, Rees CE. Taiwanese Medical Students’ narratives of intercultural professionalism dilemmas: exploring tensions between Western Medicine and Taiwanese Culture. Adv Health Sci Educ. 2017;22:429–45. https://doi.org/10.1007/s10459-016-9738-x .

Ly CL, Chun MBJ. Welcome to Cultural Competency: surgery’s efforts to Acknowledge Diversity in Residency Training. J Surg Educ. 2013;70:284–90. https://doi.org/10.1016/j.jsurg.2012.10.005 .

Crenshaw K, Shewchuk RM, Qu H, Staton LJ, Bigby JA, Houston TK, Allison J, Estrada CA. What should we include in a cultural competence curriculum? An emerging formative evaluation process to Foster Curriculum Development. Acad Med. 2011;86:333–41. https://doi.org/10.1097/ACM.0b013e3182087314 .

Minicuci N, Giorato C, Rocco I, Lloyd-Sherlock P, Avruscio G, Cardin F. Survey of doctors’ perception of Professional values. PLoS ONE. 2020;15:e0244303. https://doi.org/10.1371/journal.pone.0244303 .

Matthews M, Van Wyk J. Towards a culturally competent Health Professional: a South African case study. BMC Med Educ. 2018;18:112. https://doi.org/10.1186/s12909-018-1187-1 .

Verdon S. Awakening a critical consciousness among Multidisciplinary professionals supporting culturally and linguistically diverse families: a pilot study on the impact of Professional Development. Child Care Pract. 2020;26:4–21. https://doi.org/10.1080/13575279.2018.1516626 .

Hamdan Alshammari M, Alboliteeh M. Structural Equation Modeling of the Association between Professional and Cultural competencies of nurses in Saudi Arabia. Nurse Educ Pract. 2022;63:103382. https://doi.org/10.1016/j.nepr.2022.103382 .

Howells S, Barton G, Westerveld M. Exploring the development of Cultural Awareness amongst Post-graduate Speech-Language Pathology Students. Int J Speech-Language Pathology2016, 18, 259–71; https://doi.org/10.3109/17549507.2016.1154982

Alexis DA, Kearney MD, Williams JC, Xu C, Higginbotham EJ, Aysola J. Assessment of perceptions of professionalism among Faculty, trainees, Staff, and students in a large University-Based Health System. JAMA Netw Open. 2020;3:e2021452. https://doi.org/10.1001/jamanetworkopen.2020.21452 .

Maristany D, Hauer KE, Leep Hunderfund AN, Elks ML, Bullock JL, Kumbamu A, O’Brien BC. The Problem and Power of Professionalism: a critical analysis of medical students’ and residents’ perspectives and experiences of professionalism. Acad Med. 2023;98:S32–41. https://doi.org/10.1097/ACM.0000000000005367 .

Leyerzapf H, Abma TA, Steenwijk RR, Croiset G, Verdonk P. Standing out and moving up: Performance Appraisal of Cultural Minority Physicians. Adv Health Sci Educ. 2015;20:995–1010. https://doi.org/10.1007/s10459-014-9577-6 .

Osseo-Asare A, Balasuriya L, Huot SJ, Keene D, Berg D, Nunez-Smith M, Genao I, Latimore D, Boatright D. Minority Resident Physicians’ views on the role of Race/Ethnicity in their training experiences in the Workplace. JAMA Netw Open. 2018;1:e182723. https://doi.org/10.1001/jamanetworkopen.2018.2723 .

Kristoffersson E, Hamberg K. I have to do twice as Well – Managing Everyday Racism in a Swedish Medical School. BMC Med Educ. 2022;22:235. https://doi.org/10.1186/s12909-022-03262-5 .

Bullock JL, Lockspeiser T, Del Pino-Jones A, Richards R, Teherani A, Hauer KE. They don’t see a lot of people my color: a mixed methods study of Racial/Ethnic stereotype threat among medical students on core clerkships. Acad Med. 2020;95:S58–66. https://doi.org/10.1097/ACM.0000000000003628 .

Chandratilake M, McAleer S, Gibson J. Cultural similarities and differences in medical professionalism: a multi-region study. Med Educ. 2012;46:257–66. https://doi.org/10.1111/j.1365-2923.2011.04153.x .

Cerdeña JP, Asabor EN, Rendell S, Okolo T, Lett E. Resculpting professionalism for equity and accountability. Ann Fam Med. 2022;20:573–7. https://doi.org/10.1370/afm.2892 .

Allitt M, Frampton S. Beyond ‘Born not made’: challenging Character, emotions and Professionalism in Undergraduate Medical Education. Med Humanit. 2022;48:461–70. https://doi.org/10.1136/medhum-2021-012365 .

Rosenberg AR, Jagsi R, Marron JM. Picture a Professional: rethinking expectations of Medical Professionalism through the Lens of Diversity, Equity, and inclusion. JCO. 2021;39:4004–8. https://doi.org/10.1200/JCO.21.01500 .

AbdelHameid D. Professionalism 101 for Black Physicians. N Engl J Med. 2020;383:e34. https://doi.org/10.1056/NEJMpv2022773 .

Frye V, Camacho-Rivera M, Salas-Ramirez K, Albritton T, Deen D, Sohler N, Barrick S, Nunes J. Professionalism: the wrong Tool to solve the right problem? Acad Med. 2020;95:860–3. https://doi.org/10.1097/ACM.0000000000003266 .

Goddard VCT, Brockbank S. Re-opening Pandora’s Box: who owns professionalism and is it time for a 21st century. Definition? Med Educ. 2023;57:66–75. https://doi.org/10.1111/medu.14862 .

McKimm J, Wilkinson T. Doctors on the move: exploring professionalism in the light of cultural transitions. Med Teach. 2015;37(9):837–43. https://doi.org/10.3109/0142159X.2015.1044953 .

Wynia MK, Papadakis MA, Sullivan WM, Hafferty FW. More than a list of values and desired behaviors: a foundational understanding of medical professionalism. Acad Med. 2014;89(5):712–4. https://doi.org/10.1097/ACM.0000000000000212 .

Wang C, Hou X-Y, Khawaja NG, Dunne MP, Shakespeare-Finch J. Improvement in the cognitive aspects of Cultural competence after short-term Overseas Study Programs. IJERPH. 2021;18:7102. https://doi.org/10.3390/ijerph18137102 .

Dobson S, Voyer S, Regehr G. Perspective: agency and activism: rethinking health advocacy in the medical profession. Acad Med. 2012;87(9):1161. https://doi.org/10.1097/ACM.0b013e3182621c25 .

Sud SR, Barnert ES, Waters E, Simon P. Do medical professionalism and medical education involve commitments to political advocacy? Acad Med. 2011;86(9):1061. https://doi.org/10.1097/ACM.0b013e31822673dc .

Earnest MA, Wong SL, Federico SG. Perspective: physician advocacy: what is it and how do we do it? Acad Med. 2010;85(1):63–7. https://doi.org/10.1097/ACM.0b013e3181c40d40 .

Shaw D. Advocacy: the role of health professional associations. Int J Gynecol Obstet. 2014;127:S43–8. https://doi.org/10.1097/ACM.0b013e3181c40d40 .

Truong M, Paradies Y, Priest N. Interventions to Improve Cultural Competency in Healthcare: a systematic review of reviews. BMC Health Serv Res. 2014;14:99. https://doi.org/10.1186/1472-6963-14-99 .

Howell BA, Kristal RB, Whitmire LR, Gentry M, Rabin TL, Rosenbaum JA. Systematic Review of Advocacy Curricula in Graduate Medical Education. J GEN INTERN MED. 2019;34:2592–601. https://doi.org/10.1007/s11606-019-05184-3 .

Garcia LB, Hernandez KE, Mata H. Professional Development Through Policy Advocacy; Career Development 2015.

Pan H, Norris JL, Liang Y-S, Li J-N, Ho M-J. Building a professionalism Framework for Healthcare providers in China: a nominal group technique study. Med Teach. 2013;35:e1531–6. https://doi.org/10.3109/0142159X.2013.802299 .

Moore E, Rottmann C, Macdonald-Roach E, Making EDI, Visible. The Power of Identity Theories in Reviewing the Engineering Career Paths Literature.

Nixon SA. The coin model of privilege and critical allyship: implications for health. BMC Public Health. 2019;19(1):1637. https://doi.org/10.1186/s12889-019-7884-9 .

Boulom V, Hasanadka R, Ochoa L, Brown OW, McDevitt D, Singh TM. Importance of diversity, equity, and inclusion in the community practice setting. J Vasc Surg. 2021;74(2):S118–24. https://doi.org/10.1016/j.jvs.2021.03.051 .

Lekas HM, Pahl K, Fuller Lewis C. Rethinking cultural competence: shifting to cultural humility. Health Serv Insights. 2020;13:1178632920970580. https://doi.org/10.1177/1178632920970580 .

Sow CF, Mn C, N. V. Developing a Professionalism Education Framework at the Institutional Level withMultidisciplinary Consensus. EIMJ. 2021;13:25–40. https://doi.org/10.21315/eimj2021.13.2.3 .

Ichikawa N, Yamamoto-Mitani N, Takai Y, Tanaka M, Takemura Y. Understanding and measuring nurses’ professionalism: development and validation of the nurses’ professionalism inventory. J Nurs Adm Manag. 2020;28(7):1607–18.

Kelley KA, Stanke LD, Rabi SM, Kuba SE, Janke KK. Cross-validation of an instrument for measuring professionalism behaviors. Am J Pharm Educ. 2011;75(9):179.

Tay KT, Ng S, Hee JM, Chia EWY, Vythilingam D, Ong YT, Chiam M, Chin AMC, Fong W, Wijaya L, et al. Assessing professionalism in medicine – a scoping review of Assessment Tools from 1990 to 2018. J Med Educ Curric Dev. 2020;7:238212052095515. https://doi.org/10.1177/2382120520955159 .

Li H, Ding N, Zhang Y, Liu Y, Wen D. Assessing medical professionalism: a systematic review of instruments and their Measurement Properties. PLoS ONE. 2017;12:e0177321. https://doi.org/10.1371/journal.pone.0177321 .

Isaacson MC, Concepts. Cultural Humility or Competency. J Prof Nurs. 2014;30:251–8. https://doi.org/10.1016/j.profnurs.2013.09.011 .

Mutch J, Golden S, Purdy E, Chang CH, Oliver N, Tallentire VR. Equity, diversity and inclusion in simulation-based education: constructing a developmental framework for medical educators. Adv Simul. 2024;9(1):20.

Purdy E, Symon B, Marks RE, Speirs C, Brazil V. Exploring equity, diversity, and inclusion in a simulation program using the SIM-EDI tool: the impact of a reflexive tool for simulation educators. Adv Simul. 2023;8(1):11.

Zhou SY, Balakrishna A, Nyhof-Young J, Javeed I, Robinson LA. What do participants value in a diversity mentorship program? Perspectives from a Canadian medical school. Equality, Diversity and Inclusion: An International Journal. 2021;40(8):947 – 59.

Download references

Acknowledgements

We would like to express our gratitude to Dr. Dorothy Bakker and other members of the Michael G. DeGroote School of Medicine Professionalism Education Committee for their guidance in the development of a scoping search strategy and journal selection.

The authors conducted this study without external funding.

Author information

Authors and affiliations.

Michael G. DeGroote School of Medicine, McMaster University, Hamilton, ON, Canada

Darsh Shah, Nima Behravan & Matthew Sibbald

Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada

Nujud Al-Jabouri & Matthew Sibbald

You can also search for this author in PubMed   Google Scholar

Contributions

MS defined the research topic and study design, and as a more experienced researcher in the team, provided mentorship and constructive feedback throughout the research process. MS also guided the team in journal selection. DS, NB and NA were responsible for the development of the search strategy, devising inclusion and exclusion criteria, screening titles, abstracts and full-text articles, conducting thematic analyses and manuscript writing, with MS providing continued guidance throughout all these steps. All authors thoroughly reviewed the manuscript and provided edits and constructive feedback. The final version of the manuscript was approved by all authors. We would like to declare DS, NB and NA all as first authors with equal contributions and division of tasks throughout the research process.

Corresponding author

Correspondence to Matthew Sibbald .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Shah, D., Behravan, N., Al-Jabouri, N. et al. Incorporating equity, diversity and inclusion (EDI) into the education and assessment of professionalism for healthcare professionals and trainees: a scoping review. BMC Med Educ 24 , 991 (2024). https://doi.org/10.1186/s12909-024-05981-3

Download citation

Received : 30 April 2024

Accepted : 03 September 2024

Published : 11 September 2024

DOI : https://doi.org/10.1186/s12909-024-05981-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Scoping review
• Professionalism
• Cultural humility
• Health professions education

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

• Skip to main content
• Skip to "About this site"

Language selection

• Français
• Search and menus

Analytical Studies: Methods and References Innovation Ecosystem Performance Indicators: Review of the Literature

Skip to text

Acknowledgements

1 introduction, 2 review of innovation ecosystem indicators, 3 innovation ecosystem performance measurement framework, 4 discussion and conclusion.

Text begins

The authors would like to thank Danny Leung, Amélie Lafrance-Cooke, Ryan Kelly, Meade Conor and Catherine Beaudry for their valuable comments and suggestions.

The concept of innovation ecosystems has recently gained increasing interest among researchers, practitioners and policy makers. This article uses a systematic literature review approach by drawing on studies that bring together the most recent knowledge on innovation ecosystem performance indicators. Based on the indicators identified in these studies, the paper builds an integrated framework for analyzing the performance of innovation ecosystems based on their inputs and outputs. The results of this study allow researchers to observe the actors, activities and products that support the development of the ecosystem, including human capital, research and development, institutions, infrastructure, the business environment, financial support, products and innovation linkages, employment, and production.

As nations have made significant progress in improving their macroeconomic and institutional frameworks, attention has shifted to other engines of productivity, with the emergence of technology and innovation as core elements of the overall development process. One of the drivers of increased prosperity and economic efficiency is the extent to which an economy can adopt existing technologies or develop new technologies to improve the productivity of its industries (see, e.g. , Acs and Armington, 2004).

The concept of an innovation ecosystem has been widely discussed in the fields of strategy, innovation, and entrepreneurship, as well as among researchers, with a rapidly growing literature (see, e.g. , Gomes et al. , 2018). Researchers have developed a series of definitions related to innovation ecosystems, but a common definition would be to consider them as a dynamic set of actors (firms, universities, inventors, etc. ), activities and resources, evolving through institutions and relationships, that are important for the innovation performance of a region or country (see, e.g. , Granstrand and Holgersson, 2020).

This study explores some of the conceptual indicators used to measure the performance of innovation ecosystems. The aim of this review is to list the indicators most used in the literature on innovation ecosystems. Thus, the common framework does not suggest the construction of new indicators or the selection of ones; it describes the most widely used indicators. These indicators are based on high-quality statistics and sound, measurable analytical principles. However, this literature review considers that the choice of indicators, according to their use in the literature, may be influenced by the period of publication of the articles, or the scope (macroeconomic or microeconomic) of the articles.

This literature review on innovation ecosystems was commissioned by Statistics Canada to address the findings of the science, technology and innovation (STI) Data Gaps Initiative and to respond to the data needs of stakeholders by developing a broader, integrated measurement framework for profiling and mapping innovation ecosystems in Canada (Statistics Canada, 2021). Governments have progressively sought to adopt the concept of innovation ecosystems as a tool for promoting national and regional competitiveness, innovation, and growth. Note  1 For example, the National Research Council (NRC) has also launched a few initiatives since the early 2000s to support the growth of innovative firms clustered around NRC research institutes in different regions of Canada. In 2017, the Canadian government considered actions related to its Global Innovation Clusters in the budget, which include accelerating innovation through the provision of $950 million over five years to support several business-led innovations in superclusters.

These initiatives address the problem of Canada’s poor performance in turning its excellent research and technology into commercialized innovations (Beaudry and Solar-Pelletier, 2020). The shift to a digital economy in Canada since the COVID-19 pandemic has contributed to a surge in technology adoption through more dynamic, IT-intensive and entrepreneurial services. Note  2 It is therefore important to measure the performance of an innovation ecosystem to assess the effectiveness of the implementation of these initiatives. The performance analysis must be conducted over a given period, so the evolution of the performance over time can be observed and investment expenditures to develop innovation ecosystems can be adjusted. It is also important to be able to compare innovation ecosystems to have a precise development strategy.

The paper makes several contributions to the literature on innovation ecosystems. First, it shows how the literature measures the performance of innovation ecosystem indicators by identifying some relevant articles, which may be useful for researchers interested in identifying such measures. Second, the paper proposes a standardized and integrated framework that can be replicated in different contexts, especially in Canada. The common framework includes 8 indicator categories and 38 variables. The framework broadly follows in the footsteps of the Oslo Manual, but it differs by considering the full complexity of an innovation ecosystem and by not limiting innovation solely to the business perspective ( OECD and Eurostat, 2018). Third, the paper identifies several research opportunities based on the common innovation ecosystem performance measurement framework.

The remainder of the paper is organized as follows. Section 2 refers to a detailed literature review of innovation ecosystem indicators. Section 3 presents the common innovation ecosystem performance measurement framework. Section 4 presents a discussion of the common framework (with a synthetic example), shares opportunities for further research and concludes the study.

The concept of an innovation ecosystem is inspired by theories that study how individuals and organizations interact and collaborate formally and informally. The theoretical underpinnings of the concept cover industry clusters, as well as geographic, social and cognitive proximities, all of which have been shown to have a positive impact on a business’s propensity to innovate. A well-functioning ecosystem is necessary to increase the effectiveness of entrepreneurial and innovative activities, create jobs, and set the conditions for economic prosperity (Teece, 2007). Therefore, it is important to maintain and expand the impact of the ecosystem, which requires mapping and creating metrics to quantify and identify performance gaps, and possibly correct potential blockages (Adner, 2006).

2.1 Innovation input and output

This review proposes a common framework, illustrated in Figure 1, which the reader can use to compare and analyze the different papers studied in the following sections. These papers have analyzed innovation ecosystems in clusters, superclusters, small and large regions, and countries. The figure allows readers to examine the evolution of the frameworks proposed in the literature on innovation indicators, as well as their common features and specificities. These frameworks are built with indicators using reliable statistics and rigorous analysis methods, so that they are measurable and comparable at different geographical scales and over time. Most of these indicators help stimulate policy debates and highlight new dynamics, and this is why they are mostly developed by researchers from government agencies.

The literature on the performance of innovation ecosystems proposes indicators that can be adapted to an input–output structure. The input–output structure is based on the idea that the performance of an innovation ecosystem depends on its importance in terms of employment and production and is moderated by favourable conditions and significant investments. The inputs are the conditions that favour the creation of innovation, and outputs are the direct outcomes and indirect economic improvements that result from innovation.

The title of figure 1 is “Common Innovation Ecosystem Framework”. It gives the components of an innovation ecosystem framework.

Figure 1 consists of two main branches (top and bottom). The main branch at the top (inputs) is made up of the two conditions favoring the emergence of innovations. On the one hand (top), these are enabling conditions, i.e. human capital and research, infrastructure and institutions, and innovation links; and on the other hand (bottom), market conditions, i.e. financial support and business dynamics.

The main branch at the bottom (outputs) is made up of the two conditions accounting for the effect of innovation production. These are, on the one hand (top), innovation impacts, which are knowledge outputs; and on the other hand (bottom), economic impacts, which are employment and local production.

Source: Authors’ calculations.

Innovation inputs are divided into two groups. The first group includes enabling conditions, such as human capital, research and development (R&D), infrastructure, institutions, and innovation linkages. Human capital shapes the degree to which a country’s workforce can participate in innovative activities. Higher levels of human capital are associated with higher levels of innovation and faster technology diffusion (Heunks, 1998; Hadjimanolis, 2000; Barker and Mueller, 2002; Romijn and Albaladejo, 2002; Crescenzi, 2005; Arvanitis and Stucki, 2012). Sen (1999) describes education as a resource for more effective participation in the economic and political life of the nation. This can include improvements in education and training, which create a pool of skilled workers who can assimilate and improve imported technologies or adapt them to local conditions.

The global economy has become more sophisticated, and it is now evident that to be competitive, it is essential to enhance the human capital endowments and research capabilities of the workforce, whose members must have access to new knowledge and be continuously trained in new processes and the use of the latest technologies. The issue of the integration of a country and its private sector into the global economy has also become increasingly important over the past decade, in particular in the discussion of interactions between R&D actors.

In an increasingly interdependent regional and global economy, a more open focus on close linkages between foreign academics, entrepreneurs and innovators improves growth prospects through increased efficiency and productivity (Niosi and Bas, 2001; Santoro and Gopalakrishnan, 2001; Feller et al. , 2002; Busom and Fernández-Ribas, 2008; Eom and Lee, 2010; Soetanto and Jack, 2011; Arvanitis and Stucki, 2012). Greater integration into the economy also serves as an important channel for absorbing technological advances, including those from abroad, such as improvements in management practices, and positive effects on human capital development.

Porter (1990, 1998) provides useful information in his analysis of the role of education in bringing an economy’s productive system to full scale. He highlights the importance of close collaboration between educational institutions and potential employers, as universities and other institutions of higher learning are called upon to adapt to the changing needs of industry. An effective R&D system is therefore essential to transfer knowledge and develop innovations (see Jaffe, 1989; Audretsch and Feldman, 1996; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bode, 2004; Döring and Schnellenbach, 2006; Woodward et al. , 2006; Drucker and Goldstein, 2007; Kirchhoff et al. , 2007).

The stability of the institutional environment and the quality of infrastructure are seen as critical to private sector development. Good regulation of institutions generally means good public management and, inevitably, fewer wasted resources, the surplus of which could contribute more directly to improved productivity and growth. High-quality infrastructure, especially in machinery and equipment that integrates new technologies such as information and communications technology (ICT) assets, is essential to stimulate innovation and improve the skills of employees, thus contributing to the productivity and competitiveness of businesses (Porter, 1990, 1998; Arthurs et al. , 2009).

The second group of innovation inputs is market conditions, which include business dynamics, and finance and support. Firms develop and implement new processes that increase productivity and competitiveness by turning ideas and inventions into new goods and services that feed markets (Porter, 1990). This creates more high-value jobs and contributes to increased national wealth that can support public investments in education, health, infrastructure, and social programs. Innovation plays a key role in the competitiveness of businesses in the modern global economy. With the right knowledge and skills, entrepreneurs can better assimilate cutting-edge technologies and changing business practices, making them more likely to choose innovation-based business strategies that boost their competitiveness.

Firms benefit from a variety of research talent and financial support to design and create new knowledge, products, processes, and other innovation activities and to effectively use ICT to improve productivity. While the decision to pursue an innovation-based business strategy rests with firms, governments, and research organizations play an important role in supporting business innovation by providing financial resources, directly and indirectly. This support provides firms with an environment of reliable access to talent, knowledge, and capital to support idea development and commercialization activities (see Cheshire and Magrini, 2000; Salter and Martin, 2001; Warda, 2001; Bilbao-Osorio and Rodriguez-Pose, 2004; Wolfe and Gertler, 2004; Rodríguez-Pose and Crescenzi, 2008).

Innovation outputs are also divided into two groups. The first group is innovation impacts, which include knowledge outputs. Knowledge outputs are the observed effects of innovations. Some outputs, such as product innovations, can have a direct influence on markets, while other innovations, such as process innovations, improve the quality or marketing of services, thereby enhancing the visibility or reputation of these services ( OECD and Eurostat, 2018).

The second group of innovation outputs is economic impacts, which include employment and production. Firms’ investments in innovation are rewarded when they result in increased productivity, greater production capacity, wages, and employment, as well as increased export market share in R&D-intensive industries. Thus, firms that invest in innovation will be more profitable and contribute to the growth of the local economy. Innovation is widely regarded as a driver of productivity, which in turn is essential to higher wages, profitability for investors and improved economic welfare in the long run (Bednarzik, 2000; Kolko, 2000; Acs and Armington, 2004; Hecker, 2005; Cukier et al. , 2016).

In the next sections, the review presents studies on innovation ecosystem indicators that analyze their performance from the finest level (cluster) to the broadest level (country). See Appendix Table A.1 for an outline of these studies.

2.2 Cluster innovation ecosystem

Firms gain a competitive advantage not only from their own capabilities, but also from resources and capabilities located in the business environment geographically close to the firm. Some empirical research has shown that clustering can have significant positive effects on firm productivity, innovation, profitability, growth, and resilience (Beaudry and Breschi, 2003; Duranton and Puga, 2004; Boschma, 2005; Gordon and McCann, 2005; Martin et al. , 2011; Combes and Gobillon, 2015; Delgado and Porter, 2017). An industrial cluster refers to a group of businesses and organizations in a sector that are geographically located together, are interconnected, share common elements and are complementary to each other (Porter, 1990). This definition of clusters sets the stage for the innovation ecosystem framework used in this analysis (Granstrand and Holgersson, 2020).

The first framework analyzed is that of Arthurs et al. (2009), who propose a simple cluster framework consisting of 34 variables. They analyze the implications of this framework for the current state of eight NRC cluster initiatives by exploring some of the conceptual issues and methodological challenges encountered in the analysis of the NRC -supported clusters. Note  3 The NRC has launched several initiatives to support the growth of innovative firms clustered around their research institutes in different regions of Canada.

Much of the analytical and policy work on clusters has been based on a diverse set of quantitative measures operating at very different conceptual and spatial scales. The authors explain that STI statistics and derived indexes are inadequate in capturing the core structures and relationships that are critical to understanding the state and performance of a cluster. Indeed, these statistics do not reflect emerging technology areas, tacit knowledge, and market linkages. They are sometimes unavailable at the level of geographic disaggregation required for small clusters because of privacy restrictions. Consequently, their methodology for cluster analysis relies mainly on interviews and surveys of firms and innovators because they provide rich information on how individual clusters perform.

Their framework is built on earlier work (Porter, 1990, 1998) and incorporates the findings of the Innovation Systems Research Network regarding clusters in the Canadian context (Wolfe and Gertler, 2004). Innovation inputs consider enabling conditions and investments by firms to develop their innovation capabilities. The authors consider the conditions of the cluster environment that influence its performance, such as access to skilled human capital; current infrastructure, including the quality of transportation; and regulations that shape the business climate. Firms’ investments include support from organizations (such as NRC and government policies), customers and competitors in the development of the innovation ecosystem (here the cluster). The authors also emphasize the importance of interactions within the cluster in terms of innovation activity and add an international dimension to this interaction.

Regarding the output measures that characterize cluster performance, the authors measure cluster employment in terms of the number, size and structure of firms by adding spin-off firms. They consider the exports of firms and their growth in terms of production. They complete their framework with a measure of cluster dynamism in terms of direct innovation outputs and R&D spending. Note  4

One feature of this framework, compared with others, is that it views innovation ecosystems as a dynamically evolving system with life cycles that can be latent, developing, established or transformational. The needs and interactions of cluster actors differ according to the stage of development of the cluster, and cluster policies must evolve accordingly, according to the authors. Furthermore, compared with the present paper’s common framework, they do not consider the investments in R&D that are important to guarantee an efficient innovation capacity for the ecosystem. They also do not consider ICT assets as a useful infrastructure for innovation that can help companies and ecosystem actors generate ideas, be productive and create innovation. Although their set of indicators is relatively simple to understand, the indicators are based on opinions and generally do not generate quantitative results and are resource intensive. In addition, the authors do not explain precisely how to construct many of their variables, making replication difficult.

The next framework analyzed is that of Beaudry and Solar-Pelletier (2020), which focuses on superclusters. Superclusters are much closer to the concept of innovation ecosystems because they are generally more technology oriented.

Beyond the geographical proximity of interconnected businesses within a sector, the supercluster considers the density of a knowledge network of actors around a core technology and the ability to collaborate with customers, suppliers, and universities. The authors’ framework, from Innovation, Science and Economic Development Canada, focuses on disruptive technologies such as big data analytics, artificial intelligence, advanced materials, additive manufacturing and blockchain. This interest is also driven by the Canadian government’s Global Innovation Clusters program to resolve Canada’s innovation paradox, which highlights the difficulty of translating scientific and technological performance into effective solutions and commercial success. Supercluster partners will be able to help strengthen regional innovation ecosystems, improve the growth and competitiveness of participating businesses, and maximize economic benefits (including well-paying jobs and prosperity for Canada) through the adoption of new and potentially disruptive technologies within innovation ecosystems (ISED, 2024).

Thus, according to the authors, Canada’s innovation policy framework must be rethought to adapt to new ways of organizing and governing innovation. Their proposed framework consists of 15 variables. This framework includes only two innovation inputs, namely the firm’s R&D expenditures, to measure the firm’s commitment to innovation. However, it includes several measures of innovation output because the aim of the study was to focus on direct measures of innovation to compare different superclusters. The authors measure the innovation inputs by the number of collaborative projects or professional actors working together on the same project, between private, academic, and public organizations. They also measure business investment in R&D. The authors propose measuring innovation outputs by the capacity of firms to produce effective innovation linkages and by their ability to build the best teams and mobilize the right set of resources to foster innovation. Second, they consider a measure of knowledge output as the number of new products or processes.

Furthermore, the authors include the measures of employment and production of their innovation ecosystem in their framework. On the one hand, they look at how the ecosystem grows in terms of jobs and firms created, and high-growth firms that contribute to more than 50% of new jobs and sales in given sectors. On the other hand, they measure the output of the ecosystem by its ability to create wealth and increase economic growth and competitiveness through a rise in exports, productivity, and gross domestic product (GDP).

As mentioned above, the conditions for innovation are not developed in this framework compared with the present study’s common framework. The authors are limited to the sole investment of firms in R&D, which is not sufficient to understand the environment necessary for the creation of innovation in ecosystems. The authors recognize that this framework does not consider the human capital devoted to R&D or to tasks related to commercialization, and the mobility of the labour force in the innovation process. They also recognize the difficulties of their indicators in measuring the successful adoption of technologies that is likely to occur through informal relationships and the sharing of tacit knowledge.

2.3 Regional innovation ecosystems

The indicators in the previous frameworks are measured either at the firm level, to assess the performance of firms within clusters relative to more isolated firms, or at the cluster level, to examine the overall performance of the organizations that form a cluster. However, it is beneficial for both levels to be examined together to assess the extent to which their agreement is beneficial to the business and its environment. This is what the regional ecosystem framework allows, by promoting coherent and effective coordination of innovation that can take place at the subnational level ( e.g. , provincial, regional or city level). Thus, it enables an alternative structure of linkages and relationships beyond geographically bounded clusters.

Cukier et al. (2016) are mapping the innovation ecosystem in eastern Ontario to better understand the breadth of innovation services that can help build a competitive advantage to attract businesses and investors and help stimulate the business environment. Their framework, called the Innovation Ecosystem Scorecard, is built on an exhaustive document review, a comprehensive data analysis and discussions with key stakeholders to develop a better overall understanding of the ecosystem’s drivers, with a particular focus on the dynamics of innovation in small communities. Note  5 Since the authors are interested in innovation ecosystems in small regions, they do not look at the innovation linkages and the support and funding provided in these regions, which are generally lower, on average. They focus more on providing a wider range of indicators that can describe more important aspects of human capital, economic dynamics, productivity, and employment, and economic well-being than previous frameworks.

They build their framework on an analysis inspired by the U.S. Economic Development Administration’s framework based on business and economic development data, including job supply and demand (U.S. Economic Development Administration, 2010). This approach has allowed them to develop an inventory of key players and intermediaries in the ecosystem, including investors, large employers, incubators, business service providers and government agencies. They also use an assessment of innovation models and methods such as the Global Entrepreneurship Monitor to understand the conditions that support entrepreneurship and innovation performance, such as the availability of financing, government policies and programs, education, R&D transfer, business and physical infrastructure, and cultural and social norms. Finally, the authors supplement their study with consultations with key stakeholders to understand the components of the ecosystem and their assessment of current programs and needs.

They developed a framework consisting of 22 variables. This framework proposes a diverse set of variables whose purpose is to measure the inputs and outputs of innovation. Among the most extensive categories are (1) human capital, which examines the characteristics of the regional population and workforce (high educational attainment, young adults, and innovation-related occupations and jobs); (2) economic dynamics, which addresses local business conditions (size of existing establishments and churn of establishments); and (3) employment and output, which assesses economic growth, regional attractiveness and economic well-being (high-tech employment, population and wealth growth, worker and owner earnings, etc. ).

However, compared with the present study’s common framework, this framework does not develop (or develops very few) indicators measuring research assets within human capital or ICT assets within the infrastructure supporting innovation activities. Neither does it develop direct measures of innovation, or the resources and funds available to entrepreneurs and firms to innovate.

The framework used by Cukier and co-authors, from the U.S. Economic Development Administration, has been updated with a more extensive version. This version is presented by Slaper et al. (2016), from the Indiana Business Research Center, who propose a set of almost 70 variables that can help regional leaders reach a strong consensus on regional strategic direction in the United States. The choice of variables was based on a broad coverage of the empirical and theoretical literature. Since it is a more regional definition of the term “innovation ecosystem,” it encompasses a larger dimension of the network concept, with innovation sites, incubators, university–business research partnerships, investment capital networks and relevant workforce development programs.

The proposed indicators are more comprehensive in each of the subcategories. The framework the authors propose also uses categories based on innovation inputs and outputs to measure innovation capacity and production potential. The inputs include human capital and knowledge creation, which indicates the extent to which a region’s population can engage in innovative activities. They also include business dynamics, which are composed of a measure of firm dynamics that assesses the competitiveness of a region by tracking the entry and exit of individual firms, and a measure of firm profile that assesses local business conditions and the resources available to entrepreneurs and firms.

Outputs are divided into two categories: the employment and productivity index that describes economic growth, regional attractiveness or direct outcomes of innovative activity, and the economic well-being index that explores living standards and other economic outcomes. Compared with Cukier et al. (2016), the authors add a measure of local business conditions and resources available to entrepreneurs and firms. In addition, they add a state context and social capital index. The former identifies the financial support and useful business dynamics that help understand the innovative environment. The l identifies the regional advantages of collaborative networks that underpin a community’s ability to address its challenges. These two additions remain optional, because the theory behind them is still under development and the data are not available at fine spatial scales, especially the social capital data. Therefore, in the paper, only the first set of state context variables has been included.

The variables measuring the current level of human capital, investment in innovation, and the performance of innovation activity in terms of employment and output are more varied and consider more aspects than the previous framework. However, compared with the present study’s common framework, the authors do not consider innovation linkages; however, they briefly discuss them in the social capital index.

2.4 Country innovation ecosystem performance

This section analyzes proposed frameworks on the performance of innovation ecosystems at the country level. It places greater emphasis on the role played by governments, the infrastructure and institutional framework, and international cooperation to improve local innovation production. These studies therefore support investments to create globally competitive scale and capacity in key areas of strength and opportunity, as well as a seamless integration of organizations, activities, and funding mechanisms across the innovation ecosystem.

Cannon et al. (2015) propose an analytical framework from the Science, Technology and Innovation Council that examines 28 indicators measuring the performance of innovation ecosystems in Canada. They present an innovation ecosystem framework in which the government plays a central role. The framework proposes greater government investment with innovation support programs designed to encourage collaboration across the innovation ecosystem. Governments play an important role in supporting and encouraging business innovation. They should therefore routinely provide more direct support to high-risk, high-reward business R&D, especially in industries of economic importance to Canada. Higher education institutions should also leverage government programs to increase their research and innovation capacity.

Thus, the authors emphasize human capital, research and business resources, and innovation support and finance as important inputs for innovation. They place a strong emphasis on R&D expenditures by business enterprises that are most closely linked to product and process innovation. Increased investment in R&D must be accompanied by increased investment in other knowledge assets, including talent in the business, engineering, science, and health sectors.

The authors measure the performance of innovation ecosystems on innovation output in large businesses and in high-growth small and medium-sized enterprises (SMEs) with the potential to become major players. They recommend increasing the number of large innovative firms to improve future competitiveness and employment growth, because large firms are often more productive and tend to invest and export more than small firms.

However, their framework has little regard for ICT assets in infrastructure and institutions and accounts very little for ICT assets and the business dynamics needed to develop coherent and effective innovation capital. Moreover, with the focus on the inputs of innovation, the outputs of innovation are not well developed, especially with respect to the linkages between innovations and the effect on employment.

In their framework, similar to the present study’s common framework, Hollanders and Es-Sadki (2021) focus on the conditions that capture the key drivers of innovation performance in 27 European countries and distinguish between human resources, attractive research systems and infrastructure through digitization. They also focus on investments, which capture investments in the public and business sectors, distinguishing between funding and support, business investments, and the use of IT.

This paper follows in the footsteps of the Oslo Manual, which mainly considers innovation from the firm’s point of view ( OECD and Eurostat, 2018). Although firms play an important role in innovation, universities, non-governmental organizations, not-for-profit organizations, autonomous researchers, and others are also known to contribute to the performance of an innovation ecosystem. The common framework reflects this complex picture.

Innovation outcomes are measured by innovation activities that capture different aspects of innovation in the business sector and distinguish between innovators. They also include impacts that capture the effects of firms’ innovation activities and distinguish between impacts on employment and impacts on output, such as sales and environmental sustainability. Using this set of variables, the authors classify the innovation performance of European countries as innovation leaders, strong innovators, moderate innovators, and emerging innovators.

In addition to these variables, a contextual analysis of the impact of structural differences between countries was assessed. To better understand the differences in performance between the innovation indicators used in the main measurement framework, a set of contextual indicators was added. These contextual indicators measure differences in the performance and structure of the economy, business activities and entrepreneurship, the introduction of innovation, the institutional and legal environment, climate change performance, and demographics. These variables were added to the authors’ framework, including those that allow for international comparison, bringing the number of variables to 51. This provides the authors with a framework whose variables are distributed in a balanced way between the different categories of innovation inputs and outputs, allowing them to consider a varied set of key concepts related to innovation ecosystems.

The Organisation for Economic Co-operation and Development (OECD) (2017) has also proposed a framework similar to the present study’s common framework, with the aim of helping governments of OECD and other countries (60 countries in total) to design more effective science, innovation and industry policies in a rapidly changing digital age. The selected indicators were developed to rely on high-quality statistics and robust analytical principles and be measurable internationally, over time, and with room for improvement. This framework focuses on the impact of knowledge and digital transformation in developed country economies in the context of today’s rapidly changing digital technology landscape. It proposes a set of over 80 indicators to measure the performance of innovation ecosystems. This set of indicators is derived from previous work by academics, the OECD , Eurostat and the World Bank.

Within innovation inputs, this study focuses on human capital through knowledge, talent and skills and examines the knowledge assets that many businesses and governments see as current and future sources of long-term sustainable growth. The authors also develop indicators measuring research excellence and collaboration to help inform the policy debate through a series of metrics on the variety and nature of knowledge dissemination mechanisms in the digital age. In addition, they consider business innovation by exploring the dynamism of the business sector and the framework conditions essential for innovation.

Unlike other frameworks, this one adds leadership and competitiveness by including indicators measuring how countries seek to develop their competitive strengths and the extent to which economies can integrate and specialize along global value chains. This framework, however, is not sufficiently balanced in terms of variables in the categories of innovation inputs and outputs, compared with the previous one. The study places a strong emphasis on innovation-enabling investments, especially in ICT assets, and completely omits tangible infrastructure and innovation-enabling institutions, especially for a national study.

López-Claros and Mata (2010) developed a set of 61 variables that allowed them to construct the Innovation Capacity Index to assess how successful countries have been in developing an innovation-friendly climate capacity. The authors combine indicators used by various international organizations, including the European Commission’s Joint Research Centre, the International Monetary Fund, the OECD , the United Nations and the World Bank. This informs policy makers and entrepreneurs around the world (131 countries) about the wide range of country-specific factors that underpin innovation.

Like the previous frameworks, the authors’ framework presents indicators that can be categorized into inputs and outputs of innovation. The authors present the factors that are essential to creating an environment conducive to innovation and the types of initiatives that will contribute in some way to boosting productivity and, hence, economic growth.

In terms of innovation inputs, the authors emphasize the institutional environment that favours innovation, notably through good governance, a good country policy assessment, and a regulatory and legal framework that favours entrepreneurship. In addition, they note the importance of a good level of human capital, training, and social inclusion. Well-developed human capital resources increase the potential for innovation, which in turn increases a country’s ability to innovate and achieve sustained productivity growth. Finally, the authors stress the importance of adopting and using ICT and investing in R&D. Regarding the outputs of innovation, the emphasis is only on direct measures of innovation and knowledge products, such as patents and trademarks.

This framework is very well informed in terms of infrastructure variables and institutions. However, compared with the present study’s common framework, it lacks measures of the entrepreneurial conditions that guarantee a good innovation capability. In addition, the authors omit measures of employment and innovation linkages from their framework. Also, some underlying factors ( e.g. , budget deficit, education spending and R&D intensity) are difficult to measure, requiring surveys to capture perceptions of firms or civil society.

The World Intellectual Property Organization (WIPO) (2021) provides a framework, similar to the present study’s common framework, based on new data and analysis on the state of global innovation. This framework allows readers and policy makers to compare the performance of the innovation ecosystems of over 132 economies.

As such, the framework is designed to provide the most comprehensive picture of innovation possible, with the index comprising approximately 80 indicators, including measures of each economy’s policy environment, education, infrastructure, and knowledge creation. It has the advantage of being supported by empirical studies examining the choice of indicators as inputs and outputs of innovation. This is notably the case of Araujo Reis et al. (2021), who examine the relationship between innovation input and output with the Global Innovation Index. They show that innovation input has a significant and positive effect on innovation outputs in countries.

The WIPO framework’s translation of an economy’s investments in innovation—in the form of R&D, education, infrastructure, and strong institutions supporting innovative activities—into innovation outcomes is no small task. This study develops an effective innovation system that balances knowledge creation, exploration, and investment (the inputs of innovation) with the generation of ideas and technologies for application, exploitation and impact (the outputs of innovation).

The WIPO study is unique in that it was developed and presented during the COVID-19 pandemic, which was a trigger for innovation in economies, including the manufacture and implementation of vaccines, teleworking, and online services. According to the authors of this study, the key indicators that have been most affected by the pandemic are science, global innovation products, scientific publications, R&D expenditures, international patent filings and venture capital (VC) operations.

This section returns in more detail to the composition of the common framework proposed in Figure 1. The different studies examined above, which cover fine units such as clusters to very large units such as countries, allow more than 400 indicators of innovation performance measures to be gathered, divided between innovation inputs and outputs.

After similar indicators were combined by removing duplicates and overlaps, the indicators were ranked according to bibliometric impact importance. The nine studies presented above, as well as the sources used in these studies, are considered to justify the choice of indicators. The sources employed to validate the use of an indicator are empirical or theoretical studies that have clearly demonstrated that an indicator has an effect on innovation (innovation input) or that an indicator is affected by innovation (innovation output). The most relevant sources, cited at least 100 times in the literature, are retained. Note  6 The more a journal article is cited, the more it can be said to have influenced subsequent scientific research. This leaves 153 papers.

The indicators are then classified based on those 153 papers, from the least used to the most used. Indicators with fewer than 5 citations are weakly used, indicators with 5 to 10 citations are moderately used, and indicators with more than 10 citations are strongly used. The range of four items is the standard deviation and the mean of the distribution of the number of items per indicator. The minimum number of articles cited per indicator is 1, and the maximum number of articles cited per indicator is 20, for a total of 97 indicators listed. From the set of 97 indicators, 12 are heavily used, 26 are moderately used and 59 are lightly used in the literature. In the proposed common framework, only medium- and high-use indicators are included to reflect their role in the literature. However, on request, users can access an appendix containing all the indicators investigated in the literature in this study, to view low-use indicators and adapt them to the context of their analysis. This leaves 38 variables, as illustrated below.

Common innovation ecosystem indicators

• Education expenditures
• International students
• Knowledge programs
• Researchers and technicians in R&D
• Scientific and technical articles
• Tertiary education Note †
• University-based knowledge spillovers Note †
• Young adult population
• Broadband connections Note †
• Doing business index
• ICT investment
• Local availability of capital Note †
• Logistics performance
• Business–university collaboration
• Business local collaboration
• Business international collaboration
• Business expenditures on R&D Note †
• Business incubators Note †
• Establishment size
• High-tech industry employment Note †
• Industry concentration
• Institutionally based start-ups Note †
• International workers
• Proprietorship rate Note †
• Academic expenditures on R&D Note †
• Foreign direct investment
• Government expenditures on R&D Note †
• Gross expenditures on R&D Note †
• Venture capital investment Note †
• Patent applications Note †
• Product and process innovations
• Designs, copyrights and trademarks
• Change in establishment births Note †
• Establishment churn
• Job growth to population growth ratio
• Exports in high-tech industries Note †
• Gross domestic product growth
• Proprietor income to wages and salaries

Notes: The rest of the indicators are moderately used in the literature. R&D = research and development; ICT = information and communications technology.

Source: Authors' calculations.

The fact that the majority of indicators are weakly replicated by other studies may indicate that they either are newly used or reflect contextual objectives of researchers. For example, some recent studies consider new indicators, such as those related to the effect of innovation on environmental performance. Furthermore, the availability of a variable does not mean that a particular indicator is robust. Some variables will be more widely used because they are more widely available, introducing potential bias in the results of the analysis.

To understand the bias, two alternative classifications are proposed. The first classification includes only recent papers, from the last 10 years, reducing the number of papers analyzed from 153 to 25. Indicators with fewer than three citations are weakly used, indicators with three to five citations are moderately used and indicators with more than five citations are strongly used. The interval of two between classes is the standard deviation of the distribution of the number of papers per indicator (the mean is three). The minimum number of articles cited per indicator is 1, and the maximum number of articles cited per indicator is 8, for a total of 81 indicators listed.

The number of moderately or heavily used indicators has dropped from 38 to 32. This new ranking, as presented in Appendix B, does not change the innovation outputs in relation to the common framework. However, it does change the innovation inputs, especially the indicators measuring the quality of infrastructure and institutions, which go from five to two. The indicators of market conditions also decrease. The measures of business incubators, start-ups and public spending on R&D disappear. In sum, considering only recent studies restricts the number of indicators without adding indicators different from the common framework. The common classification is not biased by older articles.

The second classification considers the fact that macro variables are generally easier to obtain than micro analyses, and this could result in a higher number of macro studies. This would result in more macro variables, which are less suitable for an analysis of ecosystem performance at the regional level. Therefore, a bibliometric analysis was conducted using only the microanalysis of the innovation ecosystem to see whether the list of indicators selected would be different.

The new classification reduces the number of papers analyzed from 153 to 142. Indicators with fewer than five citations are weakly used, indicators with four to eight citations are moderately used and indicators with more than eight citations are strongly used. The interval of three between classes is the standard deviation of the distribution of the number of papers per indicator (the mean is four). The minimum number of articles cited per indicator is 1, and the maximum number of articles cited per indicator is 16, for a total of 56 indicators listed.

The number of moderately or heavily used indicators has dropped from 38 to 30, as presented in Appendix C. A definite bias toward macroeconomic indicators is seen. Although there are only 11 macroeconomic studies, these construct 42% of the total indicators listed, 24% of the moderately or heavily used indicators. In particular, the enabling conditions variables decrease significantly from 16 to 8. The rest of the indicators remain more or less the same, and this classification adds a new one, the measure of establishment churn, which becomes an average indicator used in the literature.

The following points briefly discuss how to measure the indicators, especially in the Canadian context. For all the indicators described here, data sources for measuring them are available upon request from Statistics Canada at the national, provincial and sometimes municipal levels.

3.1 Human capital and research

Education expenditures: Spending on education is a good indicator of the priority and level of commitment a region places on education and human capital development, which have positive implications for innovation. Education provides the basic and advanced knowledge and skills that help individuals pursue and succeed in higher education, research and employment in innovation-related fields (see López-Claros and Mata, 2010; Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). To better represent innovation, education spending can be restricted to specific areas such as tertiary education or science, technology, engineering and mathematics (STEM) programs.

International students: International students reflect the importance of academic diversity as an active channel for the dissemination of knowledge (see Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). International students can be measured as the number of students from foreign countries. This number may contain international PhD students in particular. Not all international students stay—they may choose to return to their home country after graduation. Therefore, it may be better to count the number of postgraduation work permit applicants who stay and work in Canada after graduation.

Knowledge programs: These programs indicate the quality of learning outcomes and creative thinking of human capital (see López-Claros and Mata, 2010; Cannon et al. , 2015; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). This indicator can be measured as the Programme for International Student Assessment scales in reading, mathematics and science or Programme for the International Assessment of Adult Competencies scales in numeracy, literacy and problem solving of the 15-year-old population or workers. However, this indicator is more suitable for a country innovation ecosystem.

Researchers and technicians in R&D: Researchers and other R&D personnel are an essential input to the performance of R&D. Researchers are professionals involved in the design or creation of new knowledge in business, government, higher education and private non-profit organizations. They conduct research and improve or develop ideas, models, techniques, tools, software or operating methods. The number of researchers can be weighted by population (see Porter, 1990, 1998; Wolfe and Gertler, 2004; Arthurs et al. , 2009; López-Claros and Mata, 2010; Cannon et al. , 2015; OECD , 2017; WIPO, 2021).

Scientific and technical articles: Publications are a measure of the effectiveness of the research system, as collaboration increases scientific productivity (see Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). This indicator can be measured as the number of scientific publications with at least one foreign-based co-author. It can be measured as the number of scientific and technical journal articles per million people indexed in the journal database, published in the following fields: physics, biology, chemistry, mathematics, clinical medicine, biomedical research, engineering and technology, and earth and space sciences.

Tertiary education: Government policy makers are especially interested in the supply of scientists, engineers and ICT experts because of their direct involvement in technical activities and the ongoing digital transformation (see Heunks, 1998; Hadjimanolis, 2000; Barker and Mueller, 2002; Romijn and Albaladejo, 2002; Wolfe and Gertler, 2004; Crescenzi, 2005; Arthurs et al. , 2009; Atkinson and Mayo, 2010; López-Claros and Mata, 2010; Arvanitis and Stucki, 2012; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021). STEM graduates are often employed in management positions. Tertiary education can be measured as university graduates, especially from STEM programs.

University-based knowledge spillovers: This measure estimates how scientific knowledge spreads from universities to neighbouring regions. Since universities are less competitive and profit-driven than industries, their knowledge should spread more widely between institutions and regions. This indicator could also predict the level of patenting in a region. University knowledge spillovers are measured by the distance between the university and the chosen region, or by the number of universities in cities (see Audretsch and Feldman, 1996; Anselin et al. , 1997; Breschi and Lissoni, 2001; Salter and Martin, 2001; Cohen et al. , 2002; Keller, 2002; Bode, 2004; Woodward et al. , 2006; Drucker and Goldstein, 2007; Simonen and McCann, 2008; Casper, 2013; Slaper et al. , 2016; WIPO, 2021).

Young adult population: The young population is guaranteed to include university students and working professionals, who are the most likely to engage in innovative activities. These individuals are also more likely to be less risk averse and more entrepreneurial. Moreover, the growth in the younger population suggests that new residents are likely to enhance the innovative and entrepreneurial aspects of the core community (see Slaper et al. , 2011; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; Hollanders and Es-Sadki, 2021).

3.2 Infrastructure and institutions

Broadband connections: Broadband supplies high-speed Internet connections to businesses and consumers. Several studies suggest that broadband capacity has a significant positive effect on economic performance (see Crandall et al. , 2007; López-Claros and Mata, 2010; Lehr et al. , 2006; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021). Access to high-speed Internet allows businesses and individuals to collaborate from virtually any location. Broadband connections can be measured by the number of residential broadband connections per 1,000 households.

Doing business index: This index reflects businesses’ perceptions of the ease of starting a business in their region (see Porter, 1990; Arthurs et al. , 2009; López-Claros and Mata, 2010; Hollanders and Es-Sadki, 2021; WIPO, 2021). This index would be based on the simple average of the scores for each of three indicators, such as all the procedures officially required, or commonly undertaken in practice, for an entrepreneur to formally start and operate an industrial or commercial enterprise, as well as the time and cost to complete these procedures.

ICT investment: ICT skills are highly relevant for innovation in an increasingly digital economy (see Arthurs et al. , 2009; López-Claros and Mata, 2010; Grundke et al. , 2017; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). The share of investment in ICT is an indicator of the overall development of employee skills. It can be measured in monetary terms as external IT spending (technology products purchased) and internal IT spending (custom software and training), as well as spending on telecommunications and other office equipment (as a percentage of GDP ). It can also be measured in numbers by the number of ICT specialist employees.

Local availability of capital: Local availability of capital indicates the ability of local banks to lend to businesses. Areas with a higher concentration of local bank deposits are more likely to have higher rates of entrepreneurship, innovation, new business creation and overall economic success in a region (see Porter, 1990; Wolfe and Gertler, 2004; Benfratello et al. , 2008; Arthurs et al. , 2009; Kerr and Nanda, 2009; Ayyagari et al. , 2011; Slaper et al. , 2016; WIPO, 2021). Local availability can be measured by the share of local deposits at all banks in the region, which serves as a predictor of local lending, or total capital expenditures.

Logistics performance: Logistics performance can be measured as an index assessing the perception of businesses and residents on the quality of trade and transport infrastructure and the competence and quality of logistics services (see Porter, 1990, 1998; Arthurs et al. , 2009; López-Claros and Mata, 2010; Hollanders and Es-Sadki, 2021; WIPO, 2021).

3.3 Innovation linkages

Business–university collaboration: Innovation collaboration is a platform for disseminating knowledge and accelerating innovation development (see Porter, 1998; Arthurs et al. , 2009; OECD , 2017; Beaudry and Solar-Pelletier, 2020; WIPO, 2021). Academic collaboration involves the active participation of universities and businesses. This indicator can be constructed from a survey question to businesses on the extent to which they collaborate with universities in R&D. It can also be measured by the number of research publications co-authored by universities and private businesses.

Business local collaboration: Local collaboration involves the active participation of businesses with other businesses and institutions, excluding academic organizations (see Porter, 1998; Sorenson and Fleming, 2004; Singh, 2005; Sorenson et al. , 2006; Fleming et al. , 2006; Arthurs et al. , 2009; OECD , 2017; Beaudry and Solar-Pelletier, 2020; Hollanders and Es-Sadki, 2021). This indicator can be constructed from a survey question about the number of firms (usually innovative SMEs) that had cooperation agreements on innovation activities with other firms or institutions during the recent survey period. It can also be measured by the number of research publications co-authored by private or public-private firms.

Business international collaboration: International innovation collaboration refers to active cross-border participation in innovation collaboration (see Arthurs et al. , 2009; OECD , 2017; Beaudry and Solar-Pelletier, 2020; WIPO, 2021). This indicator can be constructed from a survey question asked to businesses about the extent to which they collaborate with foreign businesses or institutions. It can also be measured by the number (or share) of co-inventions (or co-publications) in patent families with inventors located in at least two different countries.

3.4 Business dynamics

Business expenditures on R&D: Business R&D spending reflects the participation of businesses in the creation of new knowledge, which leads to greater economic growth in the region through higher patenting and productivity levels (see Audretsch and Feldman, 1996; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Gulbrandsen and Smeby, 2005; Cannon et al. , 2015; Slaper et al. , 2016; Beaudry and Solar-Pelletier, 2020; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). Business expenditures on R&D can be expressed as the share of total business R&D expenditure in GDP .

Business incubators: Incubators provide services to new businesses in the area and help them to survive and succeed, by transferring a flow of knowledge that increases their capacity for production and innovation (see Mian, 1996; Etzkowitz, 2002, 2003; Chan and Lau, 2005; Grimaldi and Grandi, 2005; Hansson et al. , 2005; Löfsten and Lindelöf, 2005; Markman et al. , 2005; Aerts et al. , 2007; Slaper et al. , 2016; Hollanders and Es-Sadki, 2021). This measure can be calculated by the proportion of business incubators in a region.

Establishment size: Small firms are highly adaptable and can easily change their processes to incorporate new ideas or technologies (see Acs and Audretsch, 1988, 1990; Porter, 1990; Arthurs et al. , 2009; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017). Large firms would contribute positively to innovation through the increased availability of funds for R&D (see Acs and Audretsch, 1988; Porter, 1990; Hicks and Hegde, 2005; Arthurs et al. , 2009; Cukier et al. , 2016; Slaper et al. , 2016). Large and small establishments can be measured as the number of establishments with fewer than 50 employees (small firms) or more than 500 employees (large firms) per 10,000 workers. However, there may not be many differences between regions in terms of small businesses per employee. It may be relevant to consider the percentage of employees in small versus large businesses, which varies more from region to region.

High-tech industry employment: Innovative areas contain businesses that require a highly skilled and specialized workforce. Employees in high tech provide services to consumers directly, such as telecommunications, and provide raw materials to innovative businesses in all sectors of the economy (see Audretsch and Feldman, 1996; Klepper, 1996; Kolko, 2000; Feser, 2003; Florida, 2003; Wolfe and Gertler, 2004; Koo, 2005; Arthurs et al. , 2009; Belussi and Sedita, 2009; Tödtling et al. , 2009; Neffke et al. , 2011; Neumark et al. , 2011; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). This is knowledge-intensive service employment, represented by the sum of those in managerial, professional and technical positions as a percentage of the total people employed.

Industry concentration: The concentration measure can help determine the extent to which a country’s industrial system is competitive or uncompetitive in different industrial subsectors. Some studies show that the market structure significantly influences the production of innovations. This market structure can be competitive or poorly diversified and concentrated (see Porter, 1998, 2000; Stuart and Sorenson, 2003; Wolfe and Gertler, 2004; Feldman et al. , 2005; Arthurs et al. , 2009; Glaeser and Kerr, 2009; Delgado et al. , 2010, 2014; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). This measure of concentration can be measured in different ways. It can be measured by the Herfindahl-Hirschman Index for the national industry, defined as the sum of the squared shares of the subsectors, usually in total manufacturing output. It can also be measured by key indicators such as the number of competitors, the relative size of competitors (larger or smaller than the respondent firm or multinationals), or qualitative measures of the intensity of competition in the firm’s market.

Institutionally based start-ups: Start-ups are actively involved in licensing technologies produced by universities and other research institutes to create new and improved goods and services. Technology transfer is an inherently innovative activity, representing the transformation of new knowledge into economic, or marketable, knowledge. A high rate of institutionally based start-ups means more concentrated innovation activity (see Porter, 1990; Almeida and Kogut, 1997; Thurik and Wennekers, 1999; Carree and Thurik, 2003; Audretsch and Keilbach, 2004; Acs and Plummer, 2005; Audretsch and Lehmann, 2005; Slaper et al. , 2016; OECD , 2017). Institutional start-ups can be measured by the number of entities that universities and other non-profit research institutions have formed.

International workers: International workers can raise the level of human capital and foster technological progress. They can increase consumption, living standards and incomes in the long run. Their presence, in particular that of international STEM workers, is correlated with high rates of entrepreneurship and innovation (see Wadhwa et al. , 2008; Hart and Acs, 2011; Kerr, 2013; Langdon et al. , 2013; Slaper et al. , 2016). International workers can be measured as the number of inbound migrant workers relative to the working-age population, particularly in STEM occupations. STEM workers are better positioned to use existing innovations and create new ones.

Proprietorship rate: High proprietorship rates are associated with greater employment growth and entrepreneurial activity conducive to innovation (see Thurik and Wennekers, 1999; Audretsch and Thurik, 2001; Thurik et al. , 2008; Romero and Martínez-Román, 2012; Acs et al. , 2013; Slaper et al. , 2016). The proprietorship rate can be measured as the number of proprietors relative to the total number of employed people. It can also be measured by the proportion of self-employment in the region.

3.5 Finance and support

Academic expenditures on R&D: R&D within universities is a good predictor of the level of patenting. It predicts knowledge transfer to the private sector and subsequent innovation (see Audretsch and Feldman, 1996; Cheshire and Magrini, 2000; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Bode, 2004; Woodward et al. , 2006; Bercovitz and Feldman, 2007; Drucker and Goldstein, 2007; Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017). It can be measured as the total expenditure on education (current expenditure, capital expenditure and transfers) as a percentage of GDP .

Foreign direct investment: Foreign direct investment increases competition and gives rise to positive externalities and technology spillovers, thereby increasing dynamic efficiency (see Sjöholm, 1999; Branstetter, 2006; Blalock and Gertler, 2008; López-Claros and Mata, 2010; Slaper et al. , 2016; Hollanders and Es-Sadki, 2021; WIPO, 2021). This indicator helps determine the extent to which foreign or domestic firms invest in the region relative to others, improves knowledge and technology transfer, describes the openness of a region’s economy and community, and provides insights into a firm’s ability to improve productivity. Foreign direct investment can be measured as the average over recent years of net inflows characterized by the sum of equity, reinvested earnings, other long-term capital, and short-term capital as reported in the balance of payments. It can be expressed as a share of GDP or can also be divided by the working-age population.

Government expenditures on R&D: Government support for R&D has been found to lead to an increase in the number of patents. There is a significant relationship between industry funding and the quality and quantity of university research. R&D funding leads to greater knowledge creation and more opportunities for innovation (see Audretsch and Feldman, 1996; Salter and Martin, 2001; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Rodríguez-Pose and Crescenzi, 2008; Slaper et al. , 2016; Hollanders and Es-Sadki, 2021). Public funding of R&D includes direct funding through instruments such as grants and public contracts and indirect support through the tax system. It is therefore the sum of government tax support for R&D spending and direct funding of business R&D spending, which can be expressed as a percentage of GDP .

Gross expenditures on R&D: R&D spending is often used to forecast innovation and economic growth, and recent indexes have included R&D spending as a measure of innovation. R&D expenditures predict patenting and productivity levels and also have a positive effect on the creation of new businesses and other measures of innovation and economic development (see Audretsch and Feldman, 1996; Salter and Martin, 2001; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Rodríguez-Pose and Crescenzi, 2008; Arthurs et al. , 2009; López-Claros and Mata, 2010; Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). Gross expenditures on R&D represent the total amount of funds spent on R&D activities across all sectors (business, higher education, federal and provincial or territorial governments, private non-profit organizations, and foreign organizations).

VC investment: VC funds are used to bring new ideas to market, bring new technology to market or develop innovative businesses (see Kortum and Lerner, 2000; Engel and Keilbach, 2007; Mann and Sager, 2007; Hirukawa and Ueda, 2011; Rin et al. , 2013; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). Higher VC activity is associated with higher levels of innovation, as measured by patents and total factor productivity growth. VC investment can be measured in dollars as the private capital raised for investment in businesses, which includes seed capital (seed plus start-up) and expansion and replacement capital. It can also be measured by the total number of VC deals. VC can be expressed as a percentage of GDP and averaged over three years.

3.6 Knowledge outputs

Patent applications: The number of patents is an established measure of regional innovation, as it predicts subsequent patent filings and the birth of new industries (see Jaffe, 1989; Jaffe et al. , 1993; Hall et al. , 2001; Agrawal and Henderson, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Corrado et al. , 2009; López-Claros and Mata, 2010; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). Patent-related knowledge is shared across networks and spreads to neighbouring regions. This indicator can be measured as the number of patent applications filed by residents.

Product and process innovations: This is the direct measure of innovations. Product innovation is the creation and introduction of a new good or service to the market or an improved version of a previous product (see Wolfe and Gertler, 2004; Arthurs et al. , 2009; Corrado et al. , 2009; Cannon et al. , 2015; OECD , 2017; Beaudry and Solar-Pelletier, 2020; Hollanders and Es-Sadki, 2021). Process innovation is the introduction of a new or improved method of production or delivery.

Designs, copyrights and trademarks: Trademarks, copyrights and designs are an important indicator of innovation, especially for the service sector, because their establishment ensures consistent product quality, productivity growth and competitiveness, which is essential for innovation growth (see Corrado et al. , 2009; López-Claros and Mata, 2010; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). This indicator can be measured as the number of individual trademarks, copyrights and designs filed.

3.7 Employment

Change in establishment births: New businesses stimulate and enhance competition among firms, which increases productivity and economic growth over the long term. Start-ups also play a key role in bringing about innovations, which often change the game, open new markets, and disrupt the status quo (see Fritsch, 2008; Neumark et al. , 2011; Hyatt and Spletzer, 2013; Criscuolo et al. , 2014; Decker et al. , 2014; Foster et al. , 2016; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). This indicator can be measured as the number of newly registered establishments per 1,000 working-age people or the average of the number of new establishments less than 1 year old divided by the total number of active establishments.

Establishment churn: Technology and knowledge requirements that have changed, or even been eliminated, offer the opportunity to create new industries, processes, and jobs. Labour turnover is an indicator of the improved employment tenure of labour force workers. Workers are moving into more desirable and better-paying jobs. Similarly, labour turnover, whether measured by the creation of new businesses or the increase in the workforce of existing businesses, is an indicator of positive economic change in the region (see Criscuolo et al. , 2014; Decker et al. , 2014; Hathaway and Litan, 2014; Cukier et al. , 2016; OECD , 2017; Slaper et al. , 2016). Establishment turnover can be measured as the ratio of establishments that increase employment (with or without adding entries) to firms that reduce employment (with or without adding exits).

Job growth to population growth ratio: Employment growth is often used to measure economic growth and as an output of innovation. High employment growth relative to population growth indicates that jobs are being created faster than people are moving into an area. The ratio measures whether employment is growing faster or slower than the general population, indicating good innovation activity, as a firm’s performance is strongly correlated with its ability to innovate (see Dowrick and Nguyen, 1989; Acs and Armington, 2004; Boarnet, 2005; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Beaudry and Solar-Pelletier, 2020).

3.8 Production

Exports in high-tech industries: Exports in high-tech industries measure technological competitiveness as the ability to commercialize the results of R&D and innovation in international markets. High-tech products are key drivers of economic growth, productivity, and welfare, and they are typically a source of high-value-added, well-paying jobs (see Porter, 1990, 1998; Wolfe and Gertler, 2004; Arthurs et al. , 2009; Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; Beaudry and Solar-Pelletier, 2020; Hollanders and Es-Sadki, 2021; WIPO, 2021). Typically, these are companies in management, scientific and technical consulting services; scientific R&D services; chemicals; basic pharmaceuticals; weapons and ammunition; computer, electronic and optical products; electrical equipment, machinery and equipment; motor vehicles; trailers and semi-trailers; other transportation equipment, excluding shipbuilding, air- and spacecraft and related machinery; or medical and dental instruments. High-tech exports can be expressed as a percentage of total trade.

GDP growth: GDP can be a measure of economic performance, because it includes labour compensation and returns to capital. Innovative products or processes are undertaken because they would increase wages or profits (see Slaper et al. , 2011; Cukier et al. , 2016; Slaper et al. , 2016; Beaudry and Solar-Pelletier, 2020; Hollanders and Es-Sadki, 2021; WIPO, 2021). Since not all companies that contribute to GDP growth are innovative or active in innovation, GDP growth of high-tech companies can be considered instead. However, GDP may give a misleading reading for Canada in recent years, given population growth. GDP per hour worked ( i.e. , labour productivity) or other measures of productivity ( e.g. , multifactor or total factor productivity growth) could therefore be considered instead.

Proprietor income to wages and salaries: This measure examines the success of entrepreneurial activity by comparing owner income to total employee wages and salaries. A high ratio suggests the presence of profitable entrepreneurial activity, which may also indicate a more dynamic and innovative economy. The data on wage earnings and owner income are based on the location of the work. This allows for a stronger relationship between innovation activities and innovation rewards based on where the innovation activities took place (see Low et al. , 2005; Wong et al. , 2005; Hessels et al. , 2008; Koellinger, 2008; Goetz and Rupasingha, 2009; Minniti and Lévesque, 2010; Slaper et al. , 2016; OECD , 2017). Proprietor income to wages and salaries can be measured as the ratio of owner income to total wages and salaries.

4.1 Methodology of the innovation framework

According to the OECD ’s Handbook on Constructing Composite Indicators: Methodology and User Guide , the two main criteria for assessing composite indicators are ease of interpretation and the transparency of the methodology used. The selected indicators meet these criteria because they are widely used in the literature and at the international level, as well as on the methodological basis of simple popularity criteria. The common framework for analyzing the performance of innovation ecosystems is based on the use of factual data. Only 5 of 38 indicators (doing business, logistics performance and the three collaboration indicators) are based on indexes built on the perceptions of businesses or civil society. The framework is composed of 26 innovation input indicators and 12 innovation output indicators. It allows innovation ecosystems to be analyzed based on a variety of indicators, measuring different dimensions of innovation.

The literature review refers to indicators used to measure the performance of innovation ecosystems. These indicators can be used to construct a composite index to compare the evolution of a city’s innovation ecosystem over time or to compare two cities. A composite index can be created from all the indicators by using a simple or weighted average (based on their medium or high ranking) of the indicator scores.

Following the methodology of the European Innovation Scoreboard (see Hollanders and Es-Sadki, 2021) is proposed. It allows a simple and efficient construction of innovation indexes comparable in time and space. This methodology uses a scoring system to build a composite index. For each indicator, a base year is identified based on the availability of data from the observed sample.

Missing data are then treated by replacing, for example, missing values with those from the previous or next available year. Then, outliers are identified and replaced by the respective maximum and minimum values observed over the entire period of the observed sample. This methodology also transforms the data if they are highly skewed. Some indicators will be expressed as percentages, while others will have values that are not limited to an upper threshold. If these values are very volatile and asymmetric, they can be replaced by the square root of the indicator value. At this stage, it is possible to identify, for each indicator, the maximum score (the highest value of the sample over the whole observed period) and the minimum score (the lowest value of the sample over the whole observed period).

The values of the indicators are finally rescaled for each period to have them from 0 to 100. To do this, a formula is applied: (value−minimum(maximum)) × 100. Finally, for each observed period, a composite synthetic innovation index can be computed as the unweighted average of the rescaled scores for all indicators, where all indicators receive the same weighting (1/38 if data are available for all 38 indicators of the common framework). The European methodology proposes expressing performance scores relative to those of the European Union, which would be equivalent to expressing, for example, the composite index of Canadian cities relative to that of Canada.

4.2 Ecosystem performance analysis

Table 1 presents the indicators for two regions, A and B. The indicator values were randomly generated between the minimum and maximum values observed for each indicator at the provincial level in Canada from 2016 to 2018. They were generated for 50 regions to construct a good sample of regions. Only two regions are presented for illustrative purposes. The scores generated are at the indicator group and subgroup level. The value column gives the observed value of the indicator. The score column gives the score generated following the methodology presented above.

In the example presented, region B (58) has better overall innovation performance than region A (49). The framework allows the performance of these two ecosystems to be analyzed in more detail. Region A performs better in terms of innovation inputs than region B (54 versus 47), mainly because of better investment in the financial environment, as well as in the human and research capital of the ecosystem. By contrast, region B performs significantly better in all areas of innovation inputs than region A (68 versus 44), translating into better overall performance in terms of innovation. Another feature that this framework highlights is that region A’s better performance in terms of investment in innovation inputs does not translate well in terms of innovation outputs. This represents the major challenge for Canada in transforming its innovation investments into knowledge products. Given that these values were generated randomly, no further interpretive analysis can be done, but the common framework would allow researchers to assess, for example, which innovation input investment profile best translates into good innovation creation or existing innovation development.

The framework allows for an analysis of important categorical indexes for the inputs and outputs of innovation. This framework does not indicate how the volume of inputs affects the quantity of outputs. The OECD recommends using indicators cautiously, with the objective of providing complementary information to already established analyses on innovation ecosystems. Thus, this framework should be combined with empirical and theoretical analysis, including informed judgments and common sense.

However, the selected indicators allow regions to be ranked according to their innovation input and output performance. Spatial and temporal analysis can be performed on specific or combined dimensions of ecosystem performance. Through the common framework, readers, and users—especially in the public domain—will be able to see, very quickly, the dimensions measured in innovation ecosystems. As a source of information, therefore, this framework can assist in the development of various policies. It can be useful for quantifying and defining numerical targets and benchmarks. For example, a comparative analysis between regions with numerical indicators that are easy to understand can be used to motivate behaviour change, because one can compare oneself to others. The United Nations Development Programme’s Human Development Index classifications have inspired development practitioners to approach economic development in a broader dimension, involving composite indicators.

The analytical framework can also help develop common goals in public debate. Indexes and associated rankings are useful tools for focusing public attention on a particular set of policy issues. When supported by detailed data, they can provide valuable information about underlying strengths and weaknesses, which can then serve as a catalyst for further policy debate and efforts to improve specific areas of expertise.

In conclusion, this study covers the literature on performance indicators for innovation ecosystems at different scales of analysis. The selected indicators are easily computable, provided that the data are available. They are also regionally and internationally comparable, thanks to the methodology used to select them.

More than 400 indicators have been explored and combined into just over 100, classified according to their occurrence in the literature. The study provides a useful framework for assessment, which will be made more effective by a greater emphasis on improving the weakly developed dimensions of innovation. It is an interesting and comprehensive tool for policy makers, researchers, the private sector, and innovation actors. However, although this ranking lists the indicators most widely used in the literature, it does not include lesser-used indicators that reflect new trends and dynamics in local economies. To visualize low-use indicators and adapt them to the context of their analysis, readers can request access to the appendix containing all the indicators in the literature examined by this study.

Common innovation ecosystem indicators (recent studies)

• University-based knowledge spillovers
• Broadband connections
• High-tech industry employment
• Proprietorship rate

Common innovation ecosystem indicators (Microeconomics studies)

• Industry concentration Note †

Acs, Z. and Armington, C. (2004). Employment growth and entrepreneurial activity in cities. Regional Studies , 38(8):911–927.

Acs, Z., Audretsch, D., and Lehmann, E. E. (2013). The knowledge spillover theory of entrepreneurship. Small Business Economics , 41:757–774.

Acs, Z. J. and Audretsch, D. B. (1988). Innovation in large and small firms: An empirical analysis. The American Economic Review , 78(4):678–690.

Acs, Z. and Audretsch, D. B. (1990). Innovation and Small Firms . MIT Press, Cambridge, MA.

Acs, Z. J. and Plummer, L. A. (2005). Penetrating the "knowledge filter" in regional economies. The Annals of Regional Science , 39:439–456.

Adner, R. (2006). Match Your Innovation Strategy To Your Innovation Ecosystem. Harvard Business Review , 84:98–107; 148.

Aerts, K., Matthyssens, P., and Vandenbempt, K. (2007). Critical role and screening practices of European business incubators. Technovation , 27(5):254–267.

Agrawal, A. and Henderson, R. (2002). Putting patents in context: Exploring knowledge transfer from MIT. Management Science , 48(1):44–60.

Almeida, P. and Kogut, B. (1997). The exploration of technological diversity and the geographic localization of innovation: Start-up firms in the semiconductor industry Small Business Economics , 9(1):21–31.

Anselin, L., Varga, A., and Acs, Z. (1997). Local geographic spillovers between university research and high technology innovations. Journal of Urban Economics , 42(3):422–448.

Araujo Reis, D. Rodrigues de Moura, F., and Machado de Aragão, I. (2021). The linkage between input and output in the innovation ecosystem. Global Journal of Human-Social Science Research .

Arthurs, D., Cassidy, E., Davis, C., and Wolfe, D. (2009). Indicators to support innovation cluster policy. International Journal of Technology Management , 464:263–279.

Arvanitis, S. and Stucki, T. (2012). What determines the innovation capability of firm founders? Industrial and Corporate Change , 21(4):1049–1084.

Atkinson, R. D. and Mayo, M. J. (2010). Refueling the U.S. innovation economy: Fresh approaches to science, technology, engineering and mathematics (STEM) education.

Audretsch, D. and Keilbach, M. (2004). Entrepreneurship capital and economic performance. Regional Studies , 38(8):949–959.

Audretsch, D. and R. Thurik (2001), "Linking Entrepreneurship to Growth" , OECD Science, Technology and Industry Working Papers , No. 2001/02, OECD Publishing, Paris, https://doi.org/10.1787/736170038056.

Audretsch, D. B. and Feldman, M. P. (1996). Innovative clusters and the industry life cycle. Review of Industrial Organization , 11(2):253–273.

Audretsch, D. B. and Lehmann, E. E. (2005). Does the knowledge spillover theory of entrepreneurship hold for regions? Research Policy , 34(8):1191–1202. Regionalization of Innovation Policy.

Ayyagari, M., Demirgüç-Kunt, A., and Maksimovic, V. (2011). Firm innovation in emerging markets: The role of finance, governance, and competition. Journal of Financial and Quantitative Analysis , 46(6):1545–1580.

Barker, V. III and Mueller, G. C. (2002). CEO characteristics and firm R&D spending. Management Science , 48(6):782–801.

Beaudry, C. and Breschi, S. (2003). Are firms in clusters really more innovative? Economics of Innovation and New Technology , 12(4):325–342.

Beaudry, C., and Solar-Pelletier, L. (2020). The Superclusters Initiative: An Opportunity to Reinforce Innovation Ecosystems. IRPP Study 79 . Montreal: Institute for research on Public Policy .

Bednarzik, R. W. (2000).  The role of entrepreneurship in U.S. and European job growth. Monthly Labor Review , 123.

Belussi, F. and Sedita, S. R. (2009). Life cycle vs. multiple path dependency in industrial districts. European Planning Studies , 17(4):505–528.

Benfratello, L., Schiantarelli, F., and Sembenelli, A. (2008). Banks and innovation: Micro-econometric evidence on Italian firms. Journal of Financial Economics , 90(2):197–217.

Bercovitz, J. E. L. and Feldman, M. P. (2007). Fishing upstream: Firm innovation strategy and university research alliances. Research Policy , 36(7):930–948.

Bilbao-Osorio and Rodriguez-Pose, A. (2004). From R&D to innovation and economic growth in the EU. Growth and Change , 35(4):434–455.

Blalock, G. and Gertler, P. J. (2008). Welfare gains from foreign direct investment through technology transfer to local suppliers. Journal of International Economics , 74(2):402–421.

Boarnet, M. G. (2005). An empirical model of intrametropolitan population and employment growth. Papers in Regional Science , 73(2):135–152.

Bode, E. (2004). The spatial pattern of localized R&D spillovers: An empirical investigation for Germany. Journal of Economic Geography , 4(1):43–64.

Boschma, R. (2005). Proximity and innovation: A critical assessment. Regional Studies , 39(1):61–74.

Bottazzi, L. and Peri, G. (2002). Innovation and spillovers in regions: Evidence from European patent data. European Economic Review , 47(4):687–710.

Branstetter, L. (2006). Is foreign direct investment a channel of knowledge spillovers? Evidence from Japan’s FDI in the United States. Journal of International Economics , 68(2):325–344.

Breschi, S. and Lissoni, F. (2001). Knowledge spillovers and local innovation systems: A critical survey. Industrial and Corporate Change , 10(4):975–1005.

Busom, I. and Fernández-Ribas, A. (2008). The impact of firm participation in R&D programmes on R&D partnerships. Research Policy , 37(2):240–257.

Cannon, E., Forest, S., Chakma, A., Fortier, S., Garbutt, D., Gupta, A., Simair, M., Kang, C.-Y., Darkes, M., Knox, K., Laflamme, R., Romero, J., Scott, J., Treurnicht, I., Gracht, P., Venne-Man, N., Verschuren, A., and Secretariate, S. (2015). Science, Technology and Innovation Council to the Government of Canada. Technical report.

Carree, M. A. and Thurik, A. R. (2003). Chapter - The impact of entrepreneurship on economic growth. In Acs, Z. J. and Audretsch, D. B., editors, Handbook of Entrepreneurship Research: An Interdisciplinary Survey and Introduction , pages 437–471. Springer, Boston, MA.

Casper, S. (2013). The spill-over theory reversed: The impact of regional economies on the commercialization of university science. Research Policy , 42(1):1313–1324.

Chan, K. F. and Lau, T. (2005). Assessing technology incubator programs in the science park: the good, the bad and the ugly. Technovation , 25(10):1215–1228.

Cheshire, P. and Magrini, S. (2000). Endogenous processes in European regional growth: Convergence and policy. Growth and Change , 31(4):455–479.

Cohen, W. M., Nelson, R. R., and Walsh, J. P. (2002). Links and impacts: The influence of public research on industrial R&D. Management Science , 48(1):1–23.

Combes, P.-P. and Gobillon, L. (2015). Chapter 5 - The empirics of agglomeration economies. In Duranton, G., Henderson, J. V., and Strange, W. C., editors, Handbook of Regional and Urban Economics , volume 5, pages 247–348. Elsevier.

Corrado, C., Hulten, C., and Sichel, D. (2009). Intangible Capital and U.S. Economic Growth. Review of Income and Wealth , 55(3):661–685.

Crandall, R., Litan, R., and Lehr, W. (2007). The effects of broadband deployment on output and employment: A cross-sectional analysis of U.S. data. Issues in Economic Policy , 6.

Crescenzi, R. (2005). Innovation and regional growth in the enlarged Europe: The role of local innovative capabilities, peripherality, and education. Growth and Change , 36(1):471–507.

Criscuolo, C., P. Gal and C. Menon (2014), "The Dynamics of Employment Growth: New Evidence from 18 Countries", OECD Science, Technology and Industry Policy Papers , No. 14, OECD Publishing, Paris.

Cukier, W., Stolarik, K., Ngwenyama, O., N.,.and Elmi, M. (2016). Mapping the Innovation Ecosystem in Eastern Ontario. Technical report, Institute for Innovation and Technology Management.

Decker, R., Haltiwanger, J., Jarmin, R., and Miranda, J. (2014). The role of entrepreneurship in US job creation and economic dynamism. Journal of Economic Perspectives , 28(3):3–24.

Delgado, M. and Porter, M. E. (2017). Clusters and the Great Recession. In DRUID Conference Paper . Technical Report.

Delgado, M., Porter, M. E., and Stern, S. (2010). Clusters and entrepreneurship. Journal of Economic Geography , 10(4):495–518.

Delgado, M., Porter, M. E., and Stern, S. (2014). Clusters, convergence, and economic performance. Research Policy , 43(10):1785–1799.

Döring, T. and Schnellenbach, J. (2006). What do we know about geographical knowledge spillovers and regional growth?: A survey of the literature. Regional Studies , 40(3):375–395.

Dowrick, S. and Nguyen, D.-T. (1989). OECD comparative economic growth 1950-85: Catch-up and convergence. The American Economic Review , 79(5):1010–1030.

Drucker, J. and Goldstein, H. (2007). Assessing the regional economic development im- pacts of universities: A review of current approaches. International Regional Science Review , 30(1):20–46.

Duranton G, Puga D. 2004. Micro-Foundations of Urban Agglomeration Economies. In Handbook of Regional and Urban Economics , ed. J. V. Henderson and J.-F. Thisse, Amsterdam: Elsevier. Vol. 4: 2063-2117.

Engel, D. and Keilbach, M. (2007). Firm-level implications of early stage venture capital investment ― An empirical investigation. Journal of Empirical Finance , 14(2):150–167.

Eom, B.-Y. and Lee, K. (2010). Determinants of industry–academy linkages and, their impact on firm performance: The case of Korea as a latecomer in knowledge industrialization. Research Policy , 39(5):625–639. Special Section on Government as Entrepreneur.

Etzkowitz, H. (2002). Incubation of incubators: Innovation as a triple helix of university- industry-government networks. Science and Public Policy , 29(2):115–128.

Etzkowitz, H. (2003). Innovation in innovation: The triple helix of university-industry- government relations. Social Science Information , 42(3):293–337.

Feldman, M., Francis, J., and Bercovitz, J. (2005). Creating a cluster while building a firm: Entrepreneurs and the formation of industrial clusters. Regional Studies , 39(1):129–141.

Feller, I., Ailes, C. P., and Roessner, J. D. (2002). Impacts of research universities on technological innovation in industry: evidence from engineering research centers. Research Policy , 31(3):457–474.

Feser, E. J. (2003). What regions do rather than make: A proposed set of knowledge-based occupation clusters. Urban Studies , 40(10):1937–1958.

Fleming, L., King, C. III, and Juda, A. (2006). Small worlds and regional innovation. Organization Science , 18.

Florida, R. (2003). Cities and the creative class. City & Community , 2(1):3–19.

Foster, L., Grim, C., and Haltiwanger, J. (2016). Reallocation in the great recession: Cleansing or not? Journal of Labor Economics , 34(S1):S293–S331.

Fritsch, M. (2008). How does new business formation affect regional development? Introduction to the special issue. Small Business Economics , 30:1–14.

Glaeser, E. L. and Kerr, W. R. (2009). Local industrial conditions and entrepreneurship: How much of the spatial distribution can we explain? Journal of Economics & Management Strategy , 18(1):623–663.

Goetz, S. J. and Rupasingha, A. (2009) . Determinants of growth in non-farm proprietor densities in the US, 1990-2000. Small Business Economics , 32(4):425–438.

Gomes, L. A. d. V., Facin, A. L. F., Salerno, M. S., and Ikenami, R. K. (2018). Unpacking the innovation ecosystem construct: Evolution, gaps and trends. Technological Forecasting and Social Change , 136:30–48.

Gordon, I. R. and McCann, P. (2005). Innovation, agglomeration, and regional development. Journal of Economic Geography , 5(5):523–543.

Granstrand, O. and Holgersson, M. (2020). Innovation ecosystems: A conceptual review and a new definition. Technovation , 90-91:102098.

Grimaldi, R. and Grandi, A. (2005). Business incubators and new venture creation: An assessment of incubating models. Technovation , 25(2):111–121.

Grundke, R., Jamet, S., Kalamova, M., Keslair, F., and Squicciarini, M. (2017). Skills and global value chains: A characterization. OECD Science, Technology and Industry Working Papers , No. 2017/05, OECD Publishing, Paris.

Gulbrandsen, M. and Smeby, J.-C. (2005). Industry funding and university professors’ research performance. Research Policy , 34(6):932–950.

Hadjimanolis, A. (2000). A resource-based view of innovativeness in small firms. Technology Analysis & Strategic Management , 12(2):263–281.

Hall, B. H., Jaffe, A. B., and Trajtenberg, M. (2001). The NBER patent citations data file: Lessons, insights and methodological tools. NBER Working Paper 8498 .

Hansson, F., Husted, K., and Vestergaard, J. (2005). Second generation science parks: From structural holes jockeys to social capital catalysts of the knowledge society. Technovation , 25(9):1039–1049.

Hart, D. M. and Acs, Z. J. (2011). High-tech immigrant entrepreneurship in the United States. Economic Development Quarterly , 25(2):116–129.

Hathaway, I. and Litan, R. E. (2014). Declining business dynamism in the United States: A look at states and metros. Economic Studies at Brookings .

Hecker, D. E. (2005). High-technology employment: A NAICS-based update. Monthly Labor Review / U.S. Department of Labor, Bureau of Labor Statistics , 128:57–72.

Hessels, J., van Gelderen, M., and Thurik, R. (2008). Entrepreneurial aspirations, motivations, and their drivers. Small Business Economics , 31:323–339.

Heunks, F. J. (1998). Innovation, creativity and success. Small Business Economics , 10(3):263–272.

Hicks, D. and Hegde, D. (2005). Highly innovative small firms in the markets for technology. Research Policy , 34(5):703–716.

Hirukawa, M. and Ueda, M. (2011). Venture capital and innovation: Which is first? Pacific Economic Review , 16(4):421–465.

Hollanders, H. and Es-Sadki, N. (2021). European Innovation Scoreboard 2021. Technical report, European Union.

Hyatt, H. R. and Spletzer, J. R. (2013). The recent decline in employment dynamics. IZA Journal of Labor Economics , 2(5).

Innovation, Science and Economic Development Canada. (2024). Global Innovation Clusters. https://ised-isde.canada.ca/site/global-innovation-clusters/en.

Jaffe, A. (1989). The real effects of academic research. American Economic Review , 79(5):957–970.

Jaffe, A. B., Trajtenberg, M., and Henderson, R. (1993). Geographic localization of knowledge spillovers as evidenced by patent citations. The Quarterly Journal of Economics , 108(3):577–598.

Keller, W. (2002). Geographic localization of international technology diffusion. American Economic Review , 92(1):120–142.

Kerr, W. and Nanda, R. (2009). Financing constraints and entrepreneurship. Handbook of Research on Innovation and Entrepreneurship .

Kerr, W. R. (2013). U.S. high-skilled immigration, innovation, and entrepreneurship: Empirical approaches and evidence. Working Paper 19377, National Bureau of Economic Research.

Kirchhoff, B. A., Newbert, S. L., Hasan, I., and Armington, C. (2007). The influence of university R&D expenditures on new business formations and employment growth. Entrepreneurship Theory and Practice , 31(4):543–559.

Klepper, S. (1996). Entry, exit, growth, and innovation over the product life cycle. The American Economic Review , 86(3):562–583.

Koellinger, P. (2008). Why are some entrepreneurs more innovative than others? Small Business Economics , 31(1):21–37.

Kolko, J. (2000). The high-tech rural renaissance? Information technology, firm size and rural employment growth. Small Business Research Summary , 201.

Koo, J. (2005). Knowledge-based industry clusters: Evidenced by geographical patterns of patents in manufacturing. Urban Studies , 42(9):1487–1505.

Kortum, S. and Lerner, J. (2000). Assessing the contribution of venture capital to innovation. The RAND Journal of Economics , 31(4):674–692.

Langdon, D., McKittrick, G., Beede, D., Khan, B., and Doms, M. (2013). STEM: Good jobs now and for the future . U.S. Department of Commerce, Washington, DC.

Lehr, W., Osorio, C., Gillett, S. E., and Sirbu, M. A. (2006). Measuring broadband’s economic impact. (ESD-WP-2006-02).

Löfsten, H. and Lindelöf, P. (2005). R&D networks and product innovation patterns― academic and non-academic new technology-based firms on Science Parks. Technovation , 25(9):1025–1037.

López-Claros, A. and Mata, Y. (2010). Policies and Institutions Underpinning Country Innovation: Results from the Innovation Capacity Index. pages 3–63.

Low, S., Henderson, J., and Weiler, S. (2005). Gauging a region’s entrepreneurial potential. Economic Review , 90:61–89.

Mann, R. J. and Sager, T. W. (2007). Patents, venture capital, and software start-ups. Research Policy , 36(2):193–208.

Markman, G. D., Phan, P. H., Balkin, D. B., and Gianiodis, P. T. (2005). Entrepreneurship and university-based technology transfer. Journal of Business Venturing , 20(2):241–263. Special issue on Science Parks and incubators.

Martin, P., Mayer, T., and Mayneris, F. (2011). Public support to clusters: A firm level study of French “local productive systems.” Regional Science and Urban Economics , 41(2):108–123.

Mian, S. A. (1996). Assessing value-added contributions of university technology business incubators to tenant firms. Research Policy , 25(3):325–335.

Minniti, M. and Lévesque, M. (2010). Entrepreneurial types and economic growth. Journal of Business Venturing , 25(3):305–314.

Neffke, F., Henning, M., Boschma, R., Lundquist, K.-J., and Olander, L.-O. (2011). The dynamics of agglomeration externalities along the life cycle of industries. Regional Studies , 45(1):49–65.

Neumark, D., Wall, B., and Zhang, J. (2011). Do small businesses create more jobs? New evidence for the United States from the National Establishment Time Series. The Review of Economics and Statistics , 93(1):16–29.

Niosi, J. and Bas, T. G. (2001). The competencies of regions ― Canada’s clusters in biotechnology. Small Business Economics , 17:31–42.

OECD (2017). OECD Science, Technology and Industry Scoreboard 2017: The digital transformation . OECD Science, Technology and Industry Scoreboard. OECD Publishing, Paris.

OECD and Eurostat (2018). Oslo Manual 2018 .

Porter, M. E. (1990). The Competitive Advantage of Nations . The Free Press, New York.

Porter, M. E. (1998). Clusters and Competition: New Agendas for Companies, Governments, and Institutions . Clusters and Competition. Harvard Business Review Books, Cambridge, MA.

Porter, M. E. (2000). Location, competition, and economic development: Local clusters in a global economy. Economic Development Quarterly , 14(1):15–34.

Rin, M. D., Hellmann, T., and Puri, M. (2013). Chapter 8 - A survey of venture capital research. volume 2 of Handbook of the Economics of Finance , pages 573–648. Elsevier.

Rodríguez-Pose, A. and Crescenzi, R. (2008). Research and development, spillovers, innovation systems, and the genesis of regional growth in Europe. Regional Studies , 42(1):51–67.

Romero, I. and Martínez-Román, J. A. (2012). Self-employment and innovation. Exploring the determinants of innovative behavior in small businesses. Research Policy , 41(1):178–189.

Romijn, H. and Albaladejo, M. (2002). Determinants of innovation capability in small electronics and software firms in southeast England. Research Policy , 31(7):1053–1067.

Salter, A. J. and Martin, B. R. (2001). The economic benefits of publicly funded basic research: A critical review. Research Policy , 30(3):509–532.

Santoro, M. D. and Gopalakrishnan, S. (2001). Relationship dynamics between university research centers and industrial firms: Their impact on technology transfer activities. The Journal of Technology Transfer , 26(1-2):163–171.

Sen, A. (1999). Development as Freedom . Alfred Knopf, New York.

Simonen, J. and McCann, P. (2008). Firm innovation: The influence of R&D cooperation and the geography of human capital inputs. Journal of Urban Economics , 64(1):146–154.

Singh, J. (2005). Collaborative networks as determinants of knowledge diffusion patterns. Management Science , 51(5):756–770.

Sjöholm, F. (1999). Technology gap, competition and spillovers from direct foreign investment: Evidence from establishment data. The Journal of Development Studies , 36(1):53–73.

Slaper, T. F., Hart, N. R., Hall, T. J., and Thompson, M. F. (2011). The index of innovation: A new tool for regional analysis. Economic Development Quarterly , 25(1):36–53.

Slaper, T., van der Does, T., Egan, P., Ortuzar, G., and Strange, R. (2016). Driving Regional Innovation: The Innovation Index 2.0. Technical report, U.S. Economic Development Administration.

Soetanto, D. P, and Jack, S. L. (2011). Business incubators and the networks of technology- based firms. The Journal of Technology Transfer , 38.

Sorenson, O. and Fleming, L. (2004). Science and the diffusion of knowledge. Research Policy , 33(10):1615–1634.

Sorenson, O., Rivkin, J. W., and Fleming, L. (2006). Complexity, networks and knowledge flow. Research Policy , 35(7):994–1017.

Statistics Canada. (2021). Science, Technology and Innovation Data Gaps Initiative. What we heard - key takeaways. Unpublished report.

Stuart, T. and Sorenson, O. (2003). The geography of opportunity: Spatial heterogeneity in founding rates and the performance of biotechnology firms. Research Policy , 32(2):229–253. Special issue on technology entrepreneurship and contact information for corresponding authors.

Teece, D. J. (2007). Explicating dynamic capabilities: The nature and microfoundations of (sustainable) enterprise performance. Strategic Management Journal , 28(13):1319–1350.

Thurik, A. R., Carree, M. A., Andrévan Stel, and Audretsch, D. B. (2008). Does self-employment reduce unemployment? Journal of Business Venturing , 23(6):673–686. The Economics of Entrepreneurship.

Thurik, R. and Wennekers, S. (1999). Linking entrepreneurship and economic growth. Small Business Economics , 13:27–55.

Tödtling, F., Lehner, P., and Kaufmann, A. (2009). Do different types of innovation rely on specific kinds of knowledge interactions? Technovation , 29(1):59–71.

U.S. Department of Commerce, Economic Development Administration (2010). A Practitioner’s Guide to Economic Development Tools for Regional Competitiveness in a Knowledge-Based Economy.

Wadhwa, V., Saxenian, A., Rissing, B., and Gereffi, G. (2008). Skilled immigration and economic growth. Applied Research in Economic Development , 5(1):6–15.

Warda, J. (2001). Measuring the value of R&D tax treatment in OECD countries. STI Review , 27:185–211.

WIPO (2021). Global Innovation Index 2021: Tracking Innovation through the COVID-19 Crisis. World Intellectual Property Organization, Geneva.

Wolfe, D. A. and Gertler, M. S. (2004). Clusters from the inside and out: local dynamics and global linkages. Urban Studies , 41(5/6):1071–1093.

Wong, P. K., Ho, Y. P., and Autio, E. (2005). Entrepreneurship, innovation and economic growth: Evidence from gem data. Small Business Economics , 24(3):335–350.

Woodward, D., Figueiredo, O., and Guimarães, P. (2006). Beyond the Silicon Valley: University R&D and high-technology location. Journal of Urban Economics , 60(1):15–32.

More information

Note of appreciation.

Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued co-operation and goodwill.

Standards of service to the public

Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, the Agency has developed standards of service which its employees observe in serving its clients.

Published by authority of the Minister responsible for Statistics Canada.

© His Majesty the King in Right of Canada as represented by the Minister of Industry, 2024

Use of this publication is governed by the Statistics Canada Open Licence Agreement .

Catalogue no. 11-633-X

Frequency: Occasional