geographical limitations in research

• Privacy Policy

Home » Delimitations in Research – Types, Examples and Writing Guide

Delimitations in Research – Types, Examples and Writing Guide

Table of Contents

Delimitations

Definition:

Delimitations refer to the specific boundaries or limitations that are set in a research study in order to narrow its scope and focus. Delimitations may be related to a variety of factors, including the population being studied, the geographical location, the time period, the research design , and the methods or tools being used to collect data .

The Importance of Delimitations in Research Studies

Here are some reasons why delimitations are important in research studies:

• Provide focus : Delimitations help researchers focus on a specific area of interest and avoid getting sidetracked by tangential topics. By setting clear boundaries, researchers can concentrate their efforts on the most relevant and significant aspects of the research question.
• Increase validity : Delimitations ensure that the research is more valid by defining the boundaries of the study. When researchers establish clear criteria for inclusion and exclusion, they can better control for extraneous variables that might otherwise confound the results.
• Improve generalizability : Delimitations help researchers determine the extent to which their findings can be generalized to other populations or contexts. By specifying the sample size, geographic region, time frame, or other relevant factors, researchers can provide more accurate estimates of the generalizability of their results.
• Enhance feasibility : Delimitations help researchers identify the resources and time required to complete the study. By setting realistic parameters, researchers can ensure that the study is feasible and can be completed within the available time and resources.
• Clarify scope: Delimitations help readers understand the scope of the research project. By explicitly stating what is included and excluded, researchers can avoid confusion and ensure that readers understand the boundaries of the study.

Types of Delimitations in Research

Here are some types of delimitations in research and their significance:

Time Delimitations

This type of delimitation refers to the time frame in which the research will be conducted. Time delimitations are important because they help to narrow down the scope of the study and ensure that the research is feasible within the given time constraints.

Geographical Delimitations

Geographical delimitations refer to the geographic boundaries within which the research will be conducted. These delimitations are significant because they help to ensure that the research is relevant to the intended population or location.

Population Delimitations

Population delimitations refer to the specific group of people that the research will focus on. These delimitations are important because they help to ensure that the research is targeted to a specific group, which can improve the accuracy of the results.

Data Delimitations

Data delimitations refer to the specific types of data that will be used in the research. These delimitations are important because they help to ensure that the data is relevant to the research question and that the research is conducted using reliable and valid data sources.

Scope Delimitations

Scope delimitations refer to the specific aspects or dimensions of the research that will be examined. These delimitations are important because they help to ensure that the research is focused and that the findings are relevant to the research question.

How to Write Delimitations

In order to write delimitations in research, you can follow these steps:

• Identify the scope of your study : Determine the extent of your research by defining its boundaries. This will help you to identify the areas that are within the scope of your research and those that are outside of it.
• Determine the time frame : Decide on the time period that your research will cover. This could be a specific period, such as a year, or it could be a general time frame, such as the last decade.
• I dentify the population : Determine the group of people or objects that your study will focus on. This could be a specific age group, gender, profession, or geographic location.
• Establish the sample size : Determine the number of participants that your study will involve. This will help you to establish the number of people you need to recruit for your study.
• Determine the variables: Identify the variables that will be measured in your study. This could include demographic information, attitudes, behaviors, or other factors.
• Explain the limitations : Clearly state the limitations of your study. This could include limitations related to time, resources, sample size, or other factors that may impact the validity of your research.
• Justify the limitations : Explain why these limitations are necessary for your research. This will help readers understand why certain factors were excluded from the study.

When to Write Delimitations in Research

Here are some situations when you may need to write delimitations in research:

• When defining the scope of the study: Delimitations help to define the boundaries of your research by specifying what is and what is not included in your study. For instance, you may delimit your study by focusing on a specific population, geographic region, time period, or research methodology.
• When addressing limitations: Delimitations can also be used to address the limitations of your research. For example, if your data is limited to a certain timeframe or geographic area, you can include this information in your delimitations to help readers understand the limitations of your findings.
• When justifying the relevance of the study : Delimitations can also help you to justify the relevance of your research. For instance, if you are conducting a study on a specific population or region, you can explain why this group or area is important and how your research will contribute to the understanding of this topic.
• When clarifying the research question or hypothesis : Delimitations can also be used to clarify your research question or hypothesis. By specifying the boundaries of your study, you can ensure that your research question or hypothesis is focused and specific.
• When establishing the context of the study : Finally, delimitations can help you to establish the context of your research. By providing information about the scope and limitations of your study, you can help readers to understand the context in which your research was conducted and the implications of your findings.

Examples of Delimitations in Research

Examples of Delimitations in Research are as follows:

Research Title : “Impact of Artificial Intelligence on Cybersecurity Threat Detection”

Delimitations :

• The study will focus solely on the use of artificial intelligence in detecting and mitigating cybersecurity threats.
• The study will only consider the impact of AI on threat detection and not on other aspects of cybersecurity such as prevention, response, or recovery.
• The research will be limited to a specific type of cybersecurity threats, such as malware or phishing attacks, rather than all types of cyber threats.
• The study will only consider the use of AI in a specific industry, such as finance or healthcare, rather than examining its impact across all industries.
• The research will only consider AI-based threat detection tools that are currently available and widely used, rather than including experimental or theoretical AI models.

Research Title: “The Effects of Social Media on Academic Performance: A Case Study of College Students”

Delimitations:

• The study will focus only on college students enrolled in a particular university.
• The study will only consider social media platforms such as Facebook, Twitter, and Instagram.
• The study will only analyze the academic performance of students based on their GPA and course grades.
• The study will not consider the impact of other factors such as student demographics, socioeconomic status, or other factors that may affect academic performance.
• The study will only use self-reported data from students, rather than objective measures of their social media usage or academic performance.

Purpose of Delimitations

Some Purposes of Delimitations are as follows:

• Focusing the research : By defining the scope of the study, delimitations help researchers to narrow down their research questions and focus on specific aspects of the topic. This allows for a more targeted and meaningful study.
• Clarifying the research scope : Delimitations help to clarify the boundaries of the research, which helps readers to understand what is and is not included in the study.
• Avoiding scope creep : Delimitations help researchers to stay focused on their research objectives and avoid being sidetracked by tangential issues or data.
• Enhancing the validity of the study : By setting clear boundaries, delimitations help to ensure that the study is valid and reliable.
• Improving the feasibility of the study : Delimitations help researchers to ensure that their study is feasible and can be conducted within the time and resources available.

Applications of Delimitations

Here are some common applications of delimitations:

• Geographic delimitations : Researchers may limit their study to a specific geographic area, such as a particular city, state, or country. This helps to narrow the focus of the study and makes it more manageable.
• Time delimitations : Researchers may limit their study to a specific time period, such as a decade, a year, or a specific date range. This can be useful for studying trends over time or for comparing data from different time periods.
• Population delimitations : Researchers may limit their study to a specific population, such as a particular age group, gender, or ethnic group. This can help to ensure that the study is relevant to the population being studied.
• Data delimitations : Researchers may limit their study to specific types of data, such as survey responses, interviews, or archival records. This can help to ensure that the study is based on reliable and relevant data.
• Conceptual delimitations : Researchers may limit their study to specific concepts or variables, such as only studying the effects of a particular treatment on a specific outcome. This can help to ensure that the study is focused and clear.

Advantages of Delimitations

Some Advantages of Delimitations are as follows:

• Helps to focus the study: Delimitations help to narrow down the scope of the research and identify specific areas that need to be investigated. This helps to focus the study and ensures that the research is not too broad or too narrow.
• Defines the study population: Delimitations can help to define the population that will be studied. This can include age range, gender, geographical location, or any other factors that are relevant to the research. This helps to ensure that the study is more specific and targeted.
• Provides clarity: Delimitations help to provide clarity about the research study. By identifying the boundaries and limitations of the research, it helps to avoid confusion and ensures that the research is more understandable.
• Improves validity: Delimitations can help to improve the validity of the research by ensuring that the study is more focused and specific. This can help to ensure that the research is more accurate and reliable.
• Reduces bias: Delimitations can help to reduce bias by limiting the scope of the research. This can help to ensure that the research is more objective and unbiased.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How to Cite Research Paper – All Formats and...

Data Collection – Methods Types and Examples

Research Paper Format – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Institutional Review Board – Application Sample...

• Open access
• Published: 15 January 2020

The role of geographic bias in knowledge diffusion: a systematic review and narrative synthesis

• Mark Skopec   ORCID: orcid.org/0000-0003-0038-3753 1 ,
• Hamdi Issa 2 ,
• Julie Reed 1 , 3 &
• Matthew Harris 1  

Research Integrity and Peer Review volume  5 , Article number:  2 ( 2020 ) Cite this article

13k Accesses

42 Citations

40 Altmetric

Metrics details

Descriptive studies examining publication rates and citation counts demonstrate a geographic skew toward high-income countries (HIC), and research from low- or middle-income countries (LMICs) is generally underrepresented. This has been suggested to be due in part to reviewers’ and editors’ preference toward HIC sources; however, in the absence of controlled studies, it is impossible to assert whether there is bias or whether variations in the quality or relevance of the articles being reviewed explains the geographic divide. This study synthesizes the evidence from randomized and controlled studies that explore geographic bias in the peer review process.

A systematic review was conducted to identify research studies that explicitly explore the role of geographic bias in the assessment of the quality of research articles. Only randomized and controlled studies were included in the review. Five databases were searched to locate relevant articles. A narrative synthesis of included articles was performed to identify common findings.

The systematic literature search yielded 3501 titles from which 12 full texts were reviewed, and a further eight were identified through searching reference lists of the full texts. Of these articles, only three were randomized and controlled studies that examined variants of geographic bias. One study found that abstracts attributed to HIC sources elicited a higher review score regarding relevance of the research and likelihood to recommend the research to a colleague, than did abstracts attributed to LIC sources. Another study found that the predicted odds of acceptance for a submission to a computer science conference were statistically significantly higher for submissions from a “Top University.” Two of the studies showed the presence of geographic bias between articles from “high” or “low” prestige institutions.

Conclusions

Two of the three included studies identified that geographic bias in some form was impacting on peer review; however, further robust, experimental evidence is needed to adequately inform practice surrounding this topic. Reviewers and researchers should nonetheless be aware of whether author and institutional characteristics are interfering in their judgement of research.

Peer Review reports

Descriptive studies observe a noticeable skew of published research toward high-income countries (HICs) and institutions of significant scientific repute [ 1 , 2 , 3 ]. Indeed, a global North-South research gap still exists, with most scientific contributions originating from the U.S, the UK, Canada, and Australia [ 4 ], and a remarkably high spatial concentration of scientific activity in Europe [ 5 ]. North America and Europe receive 42.3% and 35.3% of the world’s citations, respectively, while the total contribution of the world’s citations from Africa, South America, and Oceania is lower than 5% [ 6 ]. Citation counts increase exponentially with increasing gross domestic product (GDP) [ 7 ].

Although this may be due to scientific capability, research production, and the quality of research, it is possible that research from low-and-middle-income country (LMIC) contexts is being discounted prematurely and unfairly, due to a bias against the country from which the research originates. Many argue that a significant portion of the world is being overlooked when it comes to scientific contributions, [ 5 , 6 , 7 , 8 , 9 ]. Bias may occur at any stage in the review and publication process [ 10 ]. Heuristics, or mental shortcuts, offer a possible explanation for this skew of scientific research [ 11 , 12 ]. Research articles possess intrinsic and extrinsic cues as to their quality [ 13 ]. Intrinsic cues are attributes that cannot be changed, such as the research methods [ 13 ]. The quality may be judged, for example, on adherence to the stated methods, and the strength of the evidence in the research, i.e., its internal validity. Extrinsic cues are informal stimuli that may be used, even unwittingly, to make judgments about a given research article, most notably as it relates to its quality [ 14 ].

Country of origin (COO) effects, for example, are a specific type of extrinsic cue where the country source influences a consumer’s perception of the product [ 15 , 16 ]. COO effects can explain the association consumers make between HICs and high-quality products. Consumer preference is positively correlated with the degree of economic development of the source country [ 17 ]. In such scenarios, country development status, an extrinsic cue, is used to infer product quality. HICs evoke an image of technologically advanced societies, and in the consumer’s mind, this technological advancement is necessary to produce high-quality goods. Conversely, certain consumers associate products from LMICs with poorer quality, increased risk of bad performance, and dissatisfaction, due to the lesser degree of economic development [ 14 ]. If research articles are considered a product, albeit an intellectual one, it is possible that a COO effect may be equally elicited in research review at any stage in the publication process.

Peters and Ceci’s experiment to test the reliability of the peer review process [ 18 ] was the first to highlight this issue. By altering the authorship of 12 research papers to fictional or unknown institutions they found that only one of the 12 papers resubmitted to the same journals that had previously published them a few years earlier was accepted for publication [ 18 ]. Considering Peters and Ceci’s findings, coupled with the COO effects outlined above, it is conceivable that a similar phenomenon may be observable in the evaluation of research from LMICs as well. Just as the source of a product influences the consumer’s choice to purchase it, the geographic origin of a scientific manuscript may bias a reviewer or a reader’s opinion of the research. Extrinsic cues, such as COO (equating LMICs with low-quality research) may guide the decision-making process.

Studies using implicit association test methodology have found that unconscious bias toward research from LMICs is prevalent [ 19 ]. Recently, McGillivray et al. found that articles submitted to Nature journals are less likely to progress through the publication process if from low-prestige institutions [ 20 ]. Although studies examining citation counts show that publication and citation frequency is skewed toward HICs [ 1 , 2 , 3 , 21 ], these retrospective, descriptive studies cannot definitively address [ 22 , 23 ] whether this is due to geographic bias, because these designs do not shed light on whether consumers of research (whether editors, peer reviewers, or readers) are biased by the geographic origin of the research, as opposed to, for example, considering the relevance or quality of the research. Randomized controlled trials (RCTs) are the best way to determine whether the external cue of COO is influencing how reviewers rate research articles [ 24 ]. RCTs of the role of geographic bias could inform policy on best practice in peer review and beyond. We describe a systematic review to identify RCTs that specifically examine geographic bias in the assessment of the quality of research articles to determine its full extent in the knowledge diffusion and publication process.

Search strategy

A systematic search of bibliographic databases was performed during June 2018. No time filter was applied for the search, to not restrict the already limited research available on this topic. Databases included MEDLINE, Embase, Global Health, and PsycInfo. Health Management Information Consortium (HMIC) was searched as a source of gray literature. Additional articles were identified through hand-searching the reference lists of screened full-text articles. Authors of included full-text articles were contacted and asked about their knowledge of further relevant studies.

Search terms were identified using the SPIDER tool [ 25 ]. This tool was chosen as it has been found to have greater specificity than comparable search tools (such as PICO) in qualitative evidence synthesis [ 26 ]. Using the tool as a framework, we devised search terms for each of the different categories. The “Sample” category included terms such as “Periodicals as Topic/”, “Publications/”, and “peer review.” The “Phenomenon of Interest” included terms such as “Bias” or “Prejudice.” The “Design” category would have included terms such as “RCT” or “Randomized Controlled Trial,” but it was felt that including this term in the search strategy could serve to further limit already scarce evidence of the phenomenon we were seeking to investigate. The “Evaluation” category included terms such as “Observer variation,” “implicit,” and “explicit.” Finally, the “Research Type” category would have focused on quantitative research, but, as with the “Design” category, it was decided to omit search terms in this category. Table 1 lists search terms used for each source. The search strategies for each source can be made available upon request.

Inclusion criteria

One reviewer (MS) screened retrieved titles. Two authors (MS and MH) then independently reviewed abstracts for inclusion. A consensus was reached surrounding subsequent inclusion of reviewed abstracts. Full-text articles were reviewed by MS and MH jointly. Decision to include full texts was reached by consensus between both reviewers. Both authors assessed the quality of reviewed articles. Articles were included if they were peer reviewed publications of intervention studies where the primary outcome was a quantitative research review score (relative risks (RR) or odds ratios (OR) of acceptance) assessing the role of nationality, geographic, or institutional affiliation bias among reviewers or editors of periodical journals or other scientific publications. Secondary outcomes considered for inclusion were the categorical classification of manuscripts (recommendation for review and resubmission, acceptance for publications or outright rejection). Articles published in languages other than English were considered if titles and/or abstracts seemed relevant. In these cases, authors were contacted to obtain English-language transcripts, if possible.

Only randomized, controlled intervention studies were included to ascertain the individual-level effect of geographic bias. Articles were not considered for inclusion if they did not specifically examine an aspect of geographic bias, such as the role of institutional affiliation, COO, or a variant thereof or were non-randomized, non-intervention, or descriptive studies (such as bibliometric analyses of citation counts and citation tracking, review articles, editorials, or “letters to the editor.”) because these retrospective or descriptive studies cannot offer reliable evidence regarding individual-level biases [ 22 , 23 ]. We included studies that explored any aspect of geographic bias, i.e., local, regional, national, or international.

Data abstraction

Search results were merged using reference management software (Zotero 5.0.53) to remove duplicate records. Records were exported to a spreadsheet for screening. If deemed relevant to the scope of the review according to the inclusion criteria, or if the scope was unclear from the title, abstracts were reviewed. After identification of relevant abstracts, full-text articles were reviewed. The same screening strategy was employed for articles identified through hand-searching. Where appropriate, investigators were contacted to clarify study eligibility and to determine if they were aware of similar studies. If concerns and questions about inclusion persisted upon completion of the full-text review, these were discussed within the research team.

Data analysis

Our familiarity with the subject matter led us to anticipate that the outcome measures of included studies would be too heterogeneous to conduct a comparison using a robust meta-analysis. As such, we determined a priori that a narrative synthesis would be the most appropriate method to compare eventual findings.

Study selection and characteristics

The systematic literature search yielded 3501 titles. Upon removing duplicates, 3255 titles were screened. After screening of titles, 378 abstracts were reviewed. From these abstracts, 12 full texts were reviewed for inclusion, and a further eight were identified through searching reference lists of the full texts. Of these 20 articles, three were found to meet inclusion criteria for narrative synthesis. The three corresponding authors were contacted, but no further studies were retrieved through these means. A flowchart outlining the study selection process can be found in Fig. 1 .

Flowchart detailing the study selection process

One study assessed the within-individual variation in the evaluation of a research abstract when the COO of the abstract is changed from HIC to low-income country (LIC) [ 27 ]. Two further studies investigated other dimensions of geographic bias, but still met overall criteria for inclusion. One sought to investigate three forms of bias in detail: the Matilda effect (in which papers from male authors are evaluated more favorably), the Matthew effect (in which already-famous researchers receive most of the recognition for newly published work), and the biases resulting from the fame or the prestige or ranking of the author’s institutions [ 28 ]. Notably, within this particular study, only the third objective is of relevance to this review. The third study investigated if articles published in “high-prestige” journals (as measured by journal impact factor (IF)) elicited a more positive response from the reviewers than did articles published in “low-prestige” journals [ 29 ]. As the journals investigated were the New England Journal of Medicine (NEJM, a high IF journal), and the Southern Medical Journal (SMJ, a low IF journal), both containing an explicit geographic association in their names, we included this study in our analysis.

Themes and relationships within the data were explored, compared, and discussed. A detailed investigation of sources of variability and heterogeneity between the included studies was undertaken. Validity of studies was assessed using the risk of bias assessment [ 30 ].

Characteristics of study methodology

A summary table of the characteristics of the included studies can be seen in Table 2 . Each of the characteristics is discussed in more detail below.

Trial design

Each study included in this review used a different trial design. Harris et al. used a cross-over design, whereby each subject served as their own control [ 27 ]. The intervention involved asking clinicians to read and rate four different, previously published abstracts, fictionally attributed to either of two HIC institutions, or two LIC institutions, on two separate occasions, 4 weeks apart. One abstract was for a randomized trial of directly observed treatment, short course (DOTS) for tuberculosis (TB) treatment, one compared human immunodeficiency virus (HIV) services in maternal and child health, one was for a randomized trial for the cholesterol-lowering drug rosuvastatin, and one was a cross-sectional trial of the drug methadone in the treatment of drug addicts [ 27 ]. The author affiliations were switched between each review, so that abstracts initially attributed to HIC sources were attributed to LIC sources during the second wave of review, and vice versa.

Tomkins et al used a parallel trial design. Authors randomly assigned reviewers of an annual computer science conference to either the Single-Blind Program Committee (SBPC) or the Double-Blind Program Committee (DBPC). SBPC reviewers had access to author information, whereas DBPC reviewers did not. SBPC and DBPC members conducted their reviews simultaneously, and a predicted odds of acceptance for a list of seven covariates was generated [ 28 ]. Christakis et al. used a factorial design of all different permutations of a questionnaire for both journals and attribution status. 2 2 × 2 2 , or 16, different questionnaires were generated and randomly distributed to participants [ 29 ]. Participants were sent either an article, or an abstract, either correctly attributed to the NEJM or the SMJ, fictionally attributed to the NEJM or the SMJ, or unattributed altogether. The first article concerned a treatment of diabetic gastroparesis, the second was a cost analysis of kinetic therapy in preventing complications of stroke, the third was a randomized trial of surgery as a treatment for metastases to the brain, and the fourth examined nephrotoxicity following treatment with angiotensin-converting enzyme (ACE) inhibitors and nonsteroidal anti-inflammatory drug (NSAID) therapy [ 29 ]. Reviewers were then asked to rate the abstracts/articles in five categories on a Likert scale from 1 to 5. This generated an aggregate review score between 5 and 25 for each abstract/article.

Study population

Harris et al [ 27 ] targeted English clinicians through a Qualtrics survey platform, which consists of a curated list of individuals interested in participating in research. At baseline, 551 completed surveys were obtained. Of those, 347 (63.0%) clinicians also completed the second wave of surveys. Tomkins [ 28 ] selected participants from the program committee for the Web Science and Data Mining (WSDM) 2017 conference. A total of 983 reviewers were allocated to the SBPC, and 974 to the DBPC. These reviewers evaluated a total of 500 submissions. Christakis [ 29 ] identified subjects from the American Medical Association’s (AMA) master list of licensed physicians in the U.S, and from the master list of internists who had completed the Robert Wood Johnson (RWJ) Clinical Scholars program. A total of 399 participants were found to be eligible to receive a questionnaire. In total, 264 of 399 questionnaires (66%) were returned and analyzed by the authors.

Randomization

All studies were randomized. Harris [ 27 ] employed simple randomization, which occurred in real time through the Qualtrics survey platform, so that participants would be unaware that randomization had taken place. The other two articles included did not specify how randomization was performed [ 28 , 29 ].

All three studies included a “white lie” concerning the purpose of the study. Harris [ 27 ] and Christakis [ 29 ] were deliberate in their descriptions (describing the survey as a “speed-reading survey” to “enhance anchoring and fast-thinking,” (Harris) or citing an “[examination] of how physicians use information from the medical literature” (Christakis)), thus reducing the possibility of eliciting the types of behaviors they were seeking to investigate. Whereas Tomkins [ 28 ] noted in their call for papers that was sent to authors that “WSDM 2017 will use a combination of single-blind and double-blind review,” they did not mention how or if Program Committee (PC) members were notified of this change.

Outcome measures

Harris [ 27 ] asked participants to rate the abstracts in the categories of strength of evidence, relevance to the reader, and likelihood of recommendation to a peer. Responses were on a scale of 0–100, with 0 being not at all strong, relevant, or likely to recommend, and 100 being very strong, relevant, or likely to recommend. Mean scores and 95% confidence intervals (CIs), as well as mean difference in scores between the first and the second review were reported. The overall mean within-individual difference in rating of strength of evidence between abstracts from HIC and LIC source was 1.35 [95% CI (− 0.06–2.76)]. The rating of relevance and likelihood of recommendation to a peer between abstracts from HIC and LIC source was 4.50 [95% CI (3.16–5.83)] and 3.05 [95% CI (1.77–4.33)], respectively.

Tomkins [ 28 ] invited reviewers to rate each paper and allocate a review score. Reviewers also entered a “rank” for the paper. Reviewers then completed a textual review of the submission. The authors then conducted a regression analysis to calculate the predicted OR that a single-blind reviewer gives a positive (accept) score to a paper. Seven covariates were investigated which could modify the predicted odds. Only three (whether the single most common country among the paper’s authors was the U.S., whether the reviewer was from the same country as the first author, and whether the paper originated from one of the top 50 global computer science universities) were relevant to the purpose of this review. The predicted odds for review score prediction for “Top universities” were 1.58 [95% CI (1.09–2.29]. The predicted odds for review score prediction for “Paper from the U.S.” was 1.01 [95% CI (0.66–1.55)], and the predicted odds for review score prediction for “Same country as reviewer” was 1.15 [95% CI (0.71–1.86)].

Christakis [ 29 ] asked reviewers to assign scores to abstracts or articles in five categories. Each of the five characteristics was ranked on a scale of 1 to 5, with 1 as “strongly disagree” and 5 as “this is a good study.” Authors summed the responses in each category to create an aggregate “impression score” based on those five criteria. Mean differences in impression scores associated with attribution of an article or an abstract to the NEJM were 0.71 [95% CI (− 0.44–1.87)] and 0.50 [95% CI (− 0.87–1.87), respectively. Mean differences in impression scores associated with attribution of an article or an abstract to the SMJ were − 0.12 [95% CI (− 1.53–1.30)] and − 0.95 [95% CI (− 2.41–0.52)], respectively. Figures 2 , 3 , and 4 show summary findings for each of the included studies.

Results from Harris et al. [ 23 ]. Dotted line at 0 represents no difference in review scores. Overall mean difference in rating of strength between abstracts from HIC and LIC source was 1.35 [95% CI (− 0.06–2.76)]. Overall mean difference in rating of relevance and likelihood of recommendation to a peer between abstracts HIC and LIC source was 4.50 [95% CI (3.16–5.83)] and 3.05 [95% CI (1.77–4.33)], respectively

Results from Tomkins et al [ 24 ]. Dotted line at 1 represents no difference in odds of acceptance or rejection. The predicted odds for review score prediction for “Top universities” are 1.58 [95% CI (1.09–2.29]. The predicted odds for review score prediction for “Paper from the U.S.” are 1.01 [95% CI (0.66–1.55)]. The predicted odds for review score prediction for “Same country as reviewer” are 1.15 [95% CI (0.71–1.86)]

Results from Christakis et al. [ 25 ]. Dotted line at 0 represents no difference in impression scores. Mean differences in impression scores associated with attribution of an article or an abstract to the NEJM were 0.71 [95% CI (− 0.44–1.87)] and 0.50 [95% CI (− 0.87–1.87), respectively. Mean differences in impression scores associated with attribution of an article or an abstract to the SMJ were − 0.12 [95% CI (− 1.53–1.30)] and − 0.95 [95% CI (− 2.41–0.52)], respectively

Validity assessment

Risk of bias in individual studies was assessed using the Cochrane Collaboration’s tool for assessing risk of bias in these three included studies [ 30 ]. A summary of this assessment can be seen in Fig. 5 . An additional file shows the risk of bias assessment in more detail [see Additional file 1 ].

Risk of bias assessment. Risk of bias in each included study was assessed using the Cochrane Collaboration’s Risk of Bias Assessment tool [ 26 ]. Green indicates a low risk, yellow medium risk, and red high risk of bias. A more detailed discussion can be found in Additional file 1

Limitations of the included studies

As Fig. 5 shows, despite using randomization and controlled approaches, two of the included studies suffer a risk of bias. This limits the causal inferences that can be made from those studies. Further, the parallel and factorial study designs used by Tomkins and Christakis, respectively, do not provide for within-individual comparisons. Within-individual comparisons permit observations to be attributed to bias, as each individual serves as their own control. While randomization controls for confounding, neither Tomkins nor Christakis discusses in detail how randomization was carried out. Thus, we cannot conclude if observed differences in their respective results were in fact due to bias, or some other factor. Neither Christakis nor Tomkins measures whether the blinding was successful. Harris asked participants if they noticed a change in the abstracts between waves 1 and 2, and only three respondents (< 1% of participants) mentioned that they had [ 27 ], and these were accounted for in their adjusted results. While Christakis was likely able to maintain blinding throughout the study as well, Tomkins admits that participants in their study may have “unblinded” themselves in conversation with other PC members during the course of the conference [ 28 ].

In this systematic review, only three studies were identified to fit inclusion criteria for analysis suggesting a paucity of controlled research into the topic of geographic bias. Notwithstanding the limitations in the way the three trials we included were conducted, we found that the observation that HIC research is favored over LIC research is upheld. Therefore, on the balance of the evidence reviewed, we find that the descriptive studies have been corroborated. While descriptive studies such as the ones we cite are useful in their own right, they can only go so far in revealing explicit bias in the review and consumption of scientific literature. We find that there are few substitutes for a well-conducted, randomized, controlled crossover trial to investigate within-individual bias.

It has been largely assumed that peer review serves to improve the quality of journals [ 31 ]. But Peters and Ceci’s 1982 study was the first to call this into question [ 18 ]. Commendable progress has been made to root out some sources of bias in peer review, such as requiring the registration of clinical trials, and reporting methods for blinding and randomization [ 32 ]. However, these measures concern mostly assessment of the internal validity of the research articles. Removing information from submissions that would allow for judgments based on anything other than the quality of the research should also be strongly considered. With editors and reviewers disproportionally located in HICs, they are afforded the privileged position of “custodians” of knowledge [ 9 , 33 ]. This perpetuates the uncontested knowledge hierarchy, which relegates LICs to the rank of “recipients,” rather than producers of knowledge [ 33 , 34 ]. Preventing biases from manifesting by removing author affiliations or journal names from articles could prove useful. This is already done at the peer review level for many journal types through single, double, triple, and even quadruple-blinded approaches [ 10 ]. Journals like the British Medical Journal (BMJ) have instituted open review, where reviewers sign their reports, declare competing interests and make no further covert comments to the editors [ 35 ]. Additionally, the signed reviews are seen by the authors, along with constructively worded feedback, which can be used to resubmit a revised article [ 35 ].

Interventions should also be considered at the point of “consumption” by readers on an everyday basis. As readers harbor their own prejudices, removing information from published articles that would allow readers to judge articles based on anything other than the quality of the research should be considered. Some notable databases such as PubMed already hide author affiliation until the moment that the link is accessed and the reader is redirected to the specific journal. If geographic bias is proven to be a significant issue, then journals should explore opportunities to hide author affiliations even further to not unduly influence readers’ perceptions. The Committee on Publication Ethics (COPE) should consider developing guidance on how to address geographic bias in the peer review process, to ensure that at all stages of the publication process research is being judged based on merit alone.

Other strategies include a more decentralized, open-access, and open peer review model being employed, for instance, by F1000 [ 36 ]. Their model, which includes article submission, followed by real-time peer review and commenting on both the manuscript and the associated data, enables almost immediate, increased visibility for the research, as well as a more iterative, transparent approach to review and editing of the manuscript [ 36 ]. As pointed out by the managing director, the aim of this decentralized approach to publishing without the involvement of journals is to counteract “meaningless boundaries…that provide inappropriate and misleading metadata that is projected onto the published article,” [ 37 ]. Though journals hold a significant and valuable place in the academic community and will continue to do so, the practices employed by organizations such as F1000 may have a lasting impact on leveling the playing field between research from HICs and LICs.

Only one study [ 27 ] was able to conclusively demonstrate bias impacting on the evaluation of a research article’s relevance and one’s likelihood to recommend it to a peer, but not on the strength of the research. The two inconclusive studies [ 28 , 29 ] had a weaker study design with higher risk of bias, and so their results should be interpreted with caution. Although there is descriptive evidence to suggest that geographic bias exists in research evaluation, the few RCTs investigating this subject identified through this review suggests a pressing need for further research. In addition, there is little standardization in reporting of outcome measures, making statistical comparison between studies challenging. We therefore suggest that standardization of outcome measures (such as ORs, RRs, or standardized review scores) be considered for future investigations.

Important lessons can be drawn from the included articles to support the design of future research in this space.

Distinguishing institutional affiliation from Country of Origin

To a greater or lesser extent, COO effects are elicited by the institution name. It is reasonable to presume that high-quality research necessitates a certain level of economic development [ 8 ], and if a university will be associated to a country, and that country will be associated with a level of economic development, this in turn will imply a certain amount of scientific capability, and the possibility for producing high-quality research. Often, it is clear whether the institution can be associated to a particular country. For example, Harris et al [ 27 ] used “University of Addis Ababa, Ethiopia” as one of the LIC institutional sources for the abstracts in their study, and so the COO cue is clear, not just because the country is cited, but because Addis Ababa is clearly the capital of Ethiopia. However, they also used “Harvard University, U.S.” and although it is clear that the U.S. is the COO cue, “Harvard University” has such a strong brand recognition that for most readers it would be clear that even if used alone it would be referring to the U.S. However, had an institution been used that neither has strong brand recognition, nor obvious geographic affiliation, then the external country cue might not be as clear, and the extent to which any elicited bias was due to geographic bias would have been uncertain. Thus, even if Harris et al. [ 27 ] had not also indicated the COO (“Ethiopia” or “U.S.”, respectively) in their study, reviewers may have automatically associated “Harvard University” with high-quality research (Fig. 6 ) or linked “University of Addis Ababa” to “LIC,” a lower degree of scientific capability, and poor-quality research.

Heuristic framework. Reviewers may see “Harvard University,” and through a series of reasonable assumptions arrive at the conclusion that Harvard produces high-quality research (blue arrows). The heuristic occurs when reviewers see “Harvard University” and necessarily assume that the research is of high quality, when this may not be the case

Using just institutional affiliations can be sufficient to link a given research article to a specific country, thus eliciting the geographic bias where it is present, but some care must be taken when doing so in controlled studies. Tomkins [ 28 ] found that when reviewers were aware that the article under review originated from a top university, they were more likely to recommend it for acceptance. Although they do not explicitly draw the connection between university ranking and the country’s income status, 45 of the 50 top-ranked universities in Computer Science and Information Systems are in HICs [ 38 ], and so reviewers may have been considering country income status rather than, or as well as, institutional prestige. In other words, reviewers may have been basing their recommendation to accept a given manuscript on the COO of that manuscript, implicitly favoring those submissions from top-ranked institutions in HICs. Future controlled studies using factorial designs will be able to distinguish between the relative importance of an institution brand and country brand for eliciting the COO cue.

Effect of journal attribution

Scientific journals can be viewed as products, and as with most other products, they may elicit some geographic stimulus. Particularly if their names involve an explicit geographic identifier, they may be evaluated differently based on their COO [ 17 ]. As such, renowned journals such as the NEJM, the Journal of the American Medical Association (JAMA), or the BMJ, which originate from HICs, could lead readers to assume that they are reading “high-quality” research by virtue of the fact that they are implicitly associated with HICs. Conversely, less recognizable journals, such as the SMJ, or the African Journal of Environmental Sciences and Technology (AJEST) may not benefit from this treatment and may even be evaluated less favorably because of their geographic origin.

The role of journal attribution per se was investigated by two studies [ 27 , 28 ]. Harris and colleagues [ 27 ] concluded that there was no significant effect of the interaction between journal type (high or low IF) and country source and that changing the country source was more significant than changing the journal type. Similarly, Christakis and colleagues [ 29 ] found journal attribution played no statistically significant role in impression scores between attributed and unattributed articles and abstracts, when adjusting for other covariates. Nonetheless, studies exploring geographic bias in research evaluation need to take into account the listed journal type and whether any geographic identifier is present. The NEJM and the BMJ both have strong country cues. The Lancet has strong brand recognition as a U.K.-based publication. F1000, however, is an international consortium without a specific geographic identifier. Future controlled studies should examine the relative importance of the journal characteristics in eliciting COO cues and geographic bias.

Although geographic bias may not be restricted only to the axis of HICs versus LICs and it may occur at local, regional, and national levels, it is likely that LICs are most affected by elicited biases. In the humanities and social sciences and increasingly in the biomedical sciences, some academic institutions in HICs are beginning to re-evaluate their curricula to challenge predominantly western narratives and include more diverse voices and bodies of thought [ 39 ]. Such initiatives aim to bring non-western narratives and experiences to the fore and interrupt the continuous feedback of western superiority which is the basis to this sort of geographic bias [ 40 ].

Improving visibility of LIC research through scientific collaboration is now easier than ever before through open access publication and research collaboration [ 6 ]. However, paying for fees associated with open access publishing, and remunerating authors for their expenses, may remain a privilege enjoyed by those affiliated with institutions in HICs, thus creating another barrier to parity in publication. Collaboration can be particularly important to LICs, as it introduces new technologies and capabilities which allow for research and development to continue in the future [ 8 ], although care must be taken to ensure equitable recognition which is still predominantly benefitting researchers from the HICs [ 4 ]. Considering the ties between a nation’s scientific capability and its economic progress [ 8 ], developing research partnerships could prove to be the best way to participate in the scientific discourse [ 41 ]. In the medium term, this will empower countries through mutually beneficial partnerships [ 41 ]. A more long-term objective should be the creation of sustainable policies surrounding international development, which must include a strong focus on capacity-building and scientific collaboration between HICs and LICs/LMICs.

Limitations

This review does not comprise the universe of published literature regarding COO effects and geographic bias because our search, whilst systematic and comprehensive, involved only five major databases, and although we do not have reason to believe there will be other unidentified studies, we cannot exclude that possibility. Considering the large amount of titles retrieved (3501), a pragmatic decision was made to have only one reviewer screen retrieved titles, rather than two, as would be standard practice. This may also have led to relevant studies being missed. We settled on the inclusion criteria that we chose in an effort to identify only the most robustly conducted studies, using peer-reviewed, controlled, and randomized methods, so that comment could be reliably made on the role that explicit geographic bias plays in research review. The inevitable trade-off between breadth and specificity certainly played out in this research question, and widening the search and inclusion criteria could expand the selection of articles included. Future investigations could be expanded to include abstracts submitted to conferences, such as the Peer Review Congress and the Cochrane Colloquium.

Relying solely on articles published in English likely also resulted in additional relevant articles being overlooked. A further, more exhaustive review across multiple fields, and in several languages, is warranted. Earlier iterations of the search terms were more detailed and complex than the ones ultimately used; however, more complex combinations yielded fewer results, potentially excluding relevant articles. Therefore, a more simplistic search strategy was used, relying on screening and manual exclusion of irrelevant articles. This ensured that pertinent articles would not be inadvertently excluded by the search strategy. Nonetheless, it is possible that this simpler search strategy did not include some important keywords and subject headings. This may have led us to overlook other relevant research on the topic.

We used the Cochrane Collaboration’s tool for measuring the risk of bias because this tool is most applicable for the assessment of bias in randomized trials. However, as our focus was on non-clinical trials, the methods used in the studies included in this review may differ from the methods employed in clinical settings and as such, given the context of this paper, the tool may not be as applicable to the assessment of bias and may not appropriately reflect the true risk of bias. Nonetheless, we did find that only one of three included studies had a low risk of bias.

This systematic review identified three RCTs that investigate the role of geographic bias in research evaluation and peer review. There is strong evidence provided by one robust experimental study on the topic, suggesting evidence of geographic bias in the evaluation of medical research by English clinicians, but the methodological variety and risk of bias in the remaining studies retrieved make it challenging to draw firm conclusions regarding the extent to which geographic bias elicited from institutional affiliation or COO of authors impacts on the evaluation of research more broadly. Further RCTs are necessary to conclusively determine the effect that COO has on the evaluation of scientific research. At present, the call to address inequalities in knowledge production and publication has never been greater. By drawing attention to the role geographic bias plays in the process of knowledge diffusion, prejudice against LIC research, but also other forms of geographic bias, can be addressed and rooted out among the reviewers and editors of scientific publications, and among those who read, cite, and consume those scientific publications. Indeed, academics, editors, and journal editorial boards all have important roles to play in addressing this issue.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Angiotensin-converting enzyme

African Journal of Environmental Sciences and Technology

American Medical Association

British Medical Journal

Confidence interval

Country of origin

Committee on Publication Ethics

Double-Blind Program Committee

Directly observed treatment, short course

Gross domestic product

High-income country

Human immunodeficiency virus

Health Management Information Consortium

Impact factor

Journal of the American Medical Association

Low-income country

Lower-middle-income country

New England Journal of Medicine

Nonsteroidal anti-inflammatory drug

Program Committee

Randomized controlled trial

Relative risk

Robert Wood Johnson

Single-Blind Program Committee

Southern Medical Journal

Tuberculosis

Web Science and Data Mining

Link AM. US and non-US submissions: an analysis of reviewer bias. JAMA. 1998;280(3):246–7.

Article   Google Scholar  

Opthof T. The significance of the peer review process against the background of bias: priority ratings of reviewers and editors and the prediction of citation, the role of geographical bias. Cardiovasc Res [Internet]. 2002 Dec [cited 2018 Jul 2];56(3):339–46. Available from: https://academic.oup.com/cardiovascres/article-lookup/doi/10.1016/S0008-6363(02)00712-5 .

Ross JS, Gross CP, Desai MM, Hong Y, Grant AO, Daniels SR, et al. Effect of blinded peer review on abstract acceptance. J Am Med Assoc [Internet]. 2006;295(14):1675–80 Available from: http://jama.ama-assn.org/cgi/reprint/295/14/1675 .

Cash-Gibson L, Rojas-Gualdrón DF, Pericàs JM, Benach J. Inequalities in global health inequalities research: A 50-year bibliometric analysis (1966-2015). Ergin I, editor. PLOS ONE [Internet]. 2018 Jan 31 [cited 2018 Oct 26];13(1):e0191901. Available from: http://dx.plos.org/10.1371/journal.pone.0191901 .

Bornmann L, Leydesdorff L, Walch-Solimena C, Ettl C. Mapping excellence in the geography of science: An approach based on Scopus data. J Informetr [Internet]. 2011 Oct [cited 2018 Jul 26];5(4):537–46. Available from: http://linkinghub.elsevier.com/retrieve/pii/S1751157711000617 .

Pan RK, Kaski K, Fortunato S. World citation and collaboration networks: uncovering the role of geography in science. Sci Rep [Internet]. 2012 Dec [cited 2018 Jul 2];2(1). Available from: http://www.nature.com/articles/srep00902 .

Bornmann L, Stefaner M, de Moya Anegón F, Mutz R. What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide. J Informetr [Internet]. 2014 Jul [cited 2018 Jul 26];8(3):581–93. Available from: http://linkinghub.elsevier.com/retrieve/pii/S1751157714000479 .

Gálvez A, Maqueda M, Martínez-Bueno M, Valdivia E. Scientific Publication Trends and the Developing World: What can the volume and authorship of scientific articles tell us about scientific progress in various regions? Am Sci [Internet]. 2000;88(6):526–33 Available from: http://www.jstor.org/stable/27858122 .

Keiser J, Utzinger J, Tanner M, Singer BH. Representation of authors and editors from countries with different human development indexes in the leading literature on tropical medicine: survey of current evidence. BMJ [Internet]. 2004 22 [cited 2018 Jul 2];328(7450):1229–32. Available from: http://www.bmj.com/lookup/doi/10.1136/bmj.38069.518137.F6 .

Haffar S, Bazerbachi F, Murad MH. Peer review bias: a critical review. Mayo Clin Proc [Internet]. 2019 Apr [cited 2019 Apr 26];94(4):670–6. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0025619618307079 .

Siler K, Lee K, Bero L. Measuring the effectiveness of scientific gatekeeping. Proc Natl Acad Sci [Internet]. 2015 13 [cited 2018 Jul 2];112(2):360–5. Available from: http://www.pnas.org/lookup/doi/10.1073/pnas.1418218112 .

Bornmann L. Complex tasks and simple solutions: The use of heuristics in the evaluation of research: Letter to the Editor. J Assoc Inf Sci Technol [Internet]. 2015 Aug [cited 2018 Jul 2];66(8):1738–9. Available from: http://doi.wiley.com/10.1002/asi.23318 .

Godey B, Pederzoli D, Aiello G, Donvito R, Chan P, Oh H, et al. Brand and country-of-origin effect on consumers’ decision to purchase luxury products. J Bus Res [Internet]. 2012 Oct [cited 2018 Jul 2];65(10):1461–70. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0148296311003560 .

Verlegh PWJ, Steenkamp J-BEM. A review and meta-analysis of country-of-origin research. J Econ Psychol [Internet]. 1999 Oct [cited 2018 Jul 2];20(5):521–46. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0167487099000239 .

Srinivasan N, Jain SC, Sikand K. An experimental study of two dimensions of country-of-origin (manufacturing country and branding country) using intrinsic and extrinsic cues. Int Bus Rev [Internet]. 2004 Feb [cited 2018 Jul 2];13(1):65–82. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0969593103001070 .

Peterson RA, Jolibert AJP. A meta-analysis of country-of-origin effects. J Int Bus Stud [Internet]. 1995 Dec [cited 2018 Jun 29];26(4):883–900. Available from: http://link.springer.com/10.1057/palgrave.jibs.8490824 .

Bilkey WJ, Nes E. Country-of-origin effects on product evaluations. J Int Bus Stud [Internet]. 1982 Mar [cited 2018 Jun 29];13(1):89–100. Available from: http://link.springer.com/10.1057/palgrave.jibs.8490539 .

Peters DP, Ceci SJ. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behav Brain Sci [Internet]. 1982 Jun [cited 2018 Jul 26];5(02):187. Available from: http://www.journals.cambridge.org/abstract_S0140525X00011183 .

Harris M, Macinko J, Jimenez G, Mullachery P. Measuring the bias against low-income country research: an Implicit Association Test. Glob Health [Internet]. 2017 Dec [cited 2018 Aug 7];13(1). Available from: http://globalizationandhealth.biomedcentral.com/articles/10.1186/s12992-017-0304-y .

McGillivray B, De Ranieri E. Uptake and outcome of manuscripts in Nature journals by review model and author characteristics. Res Integr Peer Rev [Internet]. 2018 [cited 2019 Apr 26];3(1):5. Available from: https://doi.org/10.1186/s41073-018-0049-z .

Victora CG, Moreira CB. North-South relations in scientific publications: editorial racism? Monteiro CA, Barata R de CB, Goldbaum M, editors. Spec Issue Sci Prod Commun Public Health [Internet]. 2006;40(Especial):36–42. Available from: http://www.fsp.usp.br/rsp .

Schulz KF, Grimes DA. Blinding in randomised trials: hiding who got what. The Lancet [Internet]. 2002 23 [cited 2018 Jul 23];359(9307):696–700. Available from: http://www.sciencedirect.com/science/article/pii/S0140673602078169 .

Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. BMJ Evid-Based Med [Internet]. 2016 23 [cited 2018 Jul 23];ebmed-2016-110401. Available from: https://ebm.bmj.com/content/early/2016/06/23/ebmed-2016-110401 .

Kendall JM. Designing a research project: randomised controlled trials and their principles. Emerg Med J [Internet]. 2003 1 [cited 2018 Jul 24];20(2):164–8. Available from: http://emj.bmj.com/cgi/doi/10.1136/emj.20.2.164 .

Cooke A, Smith D, Booth A. Beyond PICO: The SPIDER Tool for Qualitative Evidence Synthesis. Qual Health Res [Internet]. 2012 Oct [cited 2018 Nov 6];22(10):1435–43. Available from: http://journals.sagepub.com/doi/10.1177/1049732312452938 .

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res [Internet]. 2014 Dec [cited 2019 May 29];14(1):579. Available from: https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-014-0579-0 .

Harris M, Marti J, Watt H, Bhatti Y, Macinko J, Darzi AW. Explicit bias toward high-income-country research: a randomized, blinded, crossover experiment of English clinicians. Health Aff (Millwood) [Internet]. 2017;36(11):1997–2004 Available from: https://www.healthaffairs.org/doi/full/10.1377/hlthaff.2017.0773 .

Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci U S A [Internet]. 2017;114(48):12708–13 Available from: http://www.pnas.org/content/114/48/12708.full.pdf .

Christakis DA, Saint S, Saha S, Elmore JG, Welsh DE, Baker P, et al. Do physicians judge a study by its cover? An investigation of journal attribution bias. J Clin Epidemiol. 2000;53(8):773–8.

Higgins JPT. Chapter 8: Assessing risk of bias in included studies. In: Altman DG, editor. Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons, Ltd: Chichester, UK; 2008.

Google Scholar  

Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Methodology Review Group, editor. Cochrane Database Syst Rev [Internet]. 2007 18 [cited 2018 Aug 15]; Available from: http://doi.wiley.com/10.1002/14651858.MR000016.pub3 .

Rennie D. Make peer review scientific: thirty years on from the first congress on peer review, Drummond Rennie reflects on the improvements brought about by research into the process--and calls for more [Internet]. Nature. 2016 [cited 2018 Aug 15]. Available from: http://link.galegroup.com/apps/doc/A457561946/AONE?sid=googlescholar .

de Sousa Santos B. Beyond abyssal thinking: from global lines to ecologies of knowledges. Review [Internet]. 2007;30(1):45–89 Available from: https://www.ces.uc.pt/bss/documentos/AbyssalThinking.pdf .

Abu-Saad I. Where inquiry ends: the peer review process and indigenous standpoints. Am Behav Sci [Internet]. 2008 Aug [cited 2018 Jul 26];51(12):1902–18. Available from: http://journals.sagepub.com/doi/10.1177/0002764208318939

Groves T. Is open peer review the fairest system? Yes. BMJ [Internet]. 2010 Nov 16 [cited 2018 Aug 15];341:c6424. Available from: https://www.bmj.com/content/341/bmj.c6424

How it Works - F1000Research [Internet]. [cited 2019 Apr 16]. Available from: https://f1000research.com/about

r/science - Science AMA Series: I’m Rebecca Lawrence, Managing Director of F1000Research, an Open Science publishing platform designed to turn traditional publishing models on their head. The journal is dead – discuss, and AMA [Internet]. reddit. [cited 2019 Apr 16]. Available from: https://www.reddit.com/r/science/comments/2jvpsj/science_ama_series_im_rebecca_lawrence_managing/

Computer Science & Information Systems [Internet]. Top Universities. 2018 [cited 2018 Jul 20]. Available from: https://www.topuniversities.com/university-rankings/university-subject-rankings/2018/computer-science-information-systems

UCL. Liberating the Curriculum [Internet]. Teaching & Learning. 2016 [cited 2018 Aug 20]. Available from: https://www.ucl.ac.uk/teaching-learning/education-initiatives/connected-curriculum/liberating-curriculum .

Inclusive educational design [Internet]. Imperial College London. [cited 2018 Aug 20]. Available from: http://www.imperial.ac.uk/staff/educational-development/teaching-toolkit/inclusive-learning-and-teaching/inclusive-educational-design/

González-Alcaide G, Park J, Huamaní C, Ramos JM. Dominance and leadership in research activities: Collaboration between countries of differing human development is reflected through authorship order and designation as corresponding authors in scientific publications. Ouzounis CA, editor. PLOS ONE [Internet]. 2017 Aug 8 [cited 2018 Jul 26];12(8):e0182513. Available from: http://dx.plos.org/10.1371/journal.pone.0182513 .

Download references

Acknowledgements

Not applicable.

The authors received no funding for this article. MH is supported in part by the NW London NIHR Applied Research Collaboration. Imperial College London is grateful for support from the NW London NIHR Applied Research Collaboration and the Imperial NIHR Biomedical Research Centre. The views expressed in this publication are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

Author information

Authors and affiliations.

Department of Primary Care and Public Health, Imperial College London, Reynolds Building, St Dunstan’s Road, London, W6 8RP, UK

Mark Skopec, Julie Reed & Matthew Harris

Institute of Global Health Innovation, St Mary’s Hospital, Praed Street, London, W2 INY, UK

NIHR CLAHRC North West London, 369 Fulham Road, London, SW10 9NH, UK

You can also search for this author in PubMed   Google Scholar

Contributions

MS designed the study, performed the database searches, screened, abstracted, and analyzed the data for relevant articles identified, completed the first draft, and revised subsequent drafts for important intellectual content. MH proposed the study, provided feedback on search terms and search results, screened abstracts and full texts, and revised all subsequent drafts for important intellectual content. HI and JR revised subsequent drafts for important intellectual content. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Mark Skopec .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests, additional information, publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1..

Detailed risk of bias assessment. Description of data: A more detailed discussion of the risk of bias assessment of the included studies, which resulted in the abbreviated version seen in Fig. 5 .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Skopec, M., Issa, H., Reed, J. et al. The role of geographic bias in knowledge diffusion: a systematic review and narrative synthesis. Res Integr Peer Rev 5 , 2 (2020). https://doi.org/10.1186/s41073-019-0088-0

Download citation

Received : 11 June 2019

Accepted : 10 December 2019

Published : 15 January 2020

DOI : https://doi.org/10.1186/s41073-019-0088-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Geographic bias
• Randomized controlled trials
• Systematic review
• Narrative synthesis

Research Integrity and Peer Review

ISSN: 2058-8615

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Scope and Delimitations in Research

Delimitations are the boundaries that the researcher sets in a research study, deciding what to include and what to exclude. They help to narrow down the study and make it more manageable and relevant to the research goal.

Updated on October 19, 2022

All scientific research has boundaries, whether or not the authors clearly explain them. Your study's scope and delimitations are the sections where you define the broader parameters and boundaries of your research.

The scope details what your study will explore, such as the target population, extent, or study duration. Delimitations are factors and variables not included in the study.

Scope and delimitations are not methodological shortcomings; they're always under your control. Discussing these is essential because doing so shows that your project is manageable and scientifically sound.

This article covers:

• What's meant by “scope” and “delimitations”
• Why these are integral components of every study
• How and where to actually write about scope and delimitations in your manuscript
• Examples of scope and delimitations from published studies

What is the scope in a research paper?

Simply put, the scope is the domain of your research. It describes the extent to which the research question will be explored in your study.

Articulating your study's scope early on helps you make your research question focused and realistic.

It also helps decide what data you need to collect (and, therefore, what data collection tools you need to design). Getting this right is vital for both academic articles and funding applications.

What are delimitations in a research paper?

Delimitations are those factors or aspects of the research area that you'll exclude from your research. The scope and delimitations of the study are intimately linked.

Essentially, delimitations form a more detailed and narrowed-down formulation of the scope in terms of exclusion. The delimitations explain what was (intentionally) not considered within the given piece of research.

Scope and delimitations examples

Use the following examples provided by our expert PhD editors as a reference when coming up with your own scope and delimitations.

Scope example

Your research question is, “What is the impact of bullying on the mental health of adolescents?” This topic, on its own, doesn't say much about what's being investigated.

The scope, for example, could encompass:

• Variables: “bullying” (dependent variable), “mental health” (independent variable), and ways of defining or measuring them
• Bullying type: Both face-to-face and cyberbullying
• Target population: Adolescents aged 12–17
• Geographical coverage: France or only one specific town in France

Delimitations example

Look back at the previous example.

Exploring the adverse effects of bullying on adolescents' mental health is a preliminary delimitation. This one was chosen from among many possible research questions (e.g., the impact of bullying on suicide rates, or children or adults).

Delimiting factors could include:

• Research design : Mixed-methods research, including thematic analysis of semi-structured interviews and statistical analysis of a survey
• Timeframe : Data collection to run for 3 months
• Population size : 100 survey participants; 15 interviewees
• Recruitment of participants : Quota sampling (aiming for specific portions of men, women, ethnic minority students etc.)

We can see that every choice you make in planning and conducting your research inevitably excludes other possible options.

What's the difference between limitations and delimitations?

Delimitations and limitations are entirely different, although they often get mixed up. These are the main differences:

This chart explains the difference between delimitations and limitations. Delimitations are the boundaries of the study while the limitations are the characteristics of the research design or methodology.

Delimitations encompass the elements outside of the boundaries you've set and depends on your decision of what yo include and exclude. On the flip side, limitations are the elements outside of your control, such as:

• limited financial resources
• unplanned work or expenses
• unexpected events (for example, the COVID-19 pandemic)
• time constraints
• lack of technology/instruments
• unavailable evidence or previous research on the topic

Delimitations involve narrowing your study to make it more manageable and relevant to what you're trying to prove. Limitations influence the validity and reliability of your research findings. Limitations are seen as potential weaknesses in your research.

Example of the differences

To clarify these differences, go back to the limitations of the earlier example.

Limitations could comprise:

• Sample size : Not large enough to provide generalizable conclusions.
• Sampling approach : Non-probability sampling has increased bias risk. For instance, the researchers might not manage to capture the experiences of ethnic minority students.
• Methodological pitfalls : Research participants from an urban area (Paris) are likely to be more advantaged than students in rural areas. A study exploring the latter's experiences will probably yield very different findings.

Where do you write the scope and delimitations, and why?

It can be surprisingly empowering to realize you're restricted when conducting scholarly research. But this realization also makes writing up your research easier to grasp and makes it easier to see its limits and the expectations placed on it. Properly revealing this information serves your field and the greater scientific community.

Openly (but briefly) acknowledge the scope and delimitations of your study early on. The Abstract and Introduction sections are good places to set the parameters of your paper.

Next, discuss the scope and delimitations in greater detail in the Methods section. You'll need to do this to justify your methodological approach and data collection instruments, as well as analyses

At this point, spell out why these delimitations were set. What alternative options did you consider? Why did you reject alternatives? What could your study not address?

Let's say you're gathering data that can be derived from different but related experiments. You must convince the reader that the one you selected best suits your research question.

Finally, a solid paper will return to the scope and delimitations in the Findings or Discussion section. Doing so helps readers contextualize and interpret findings because the study's scope and methods influence the results.

For instance, agricultural field experiments carried out under irrigated conditions yield different results from experiments carried out without irrigation.

Being transparent about the scope and any outstanding issues increases your research's credibility and objectivity. It helps other researchers replicate your study and advance scientific understanding of the same topic (e.g., by adopting a different approach).

How do you write the scope and delimitations?

Define the scope and delimitations of your study before collecting data. This is critical. This step should be part of your research project planning.

Answering the following questions will help you address your scope and delimitations clearly and convincingly.

• What are your study's aims and objectives?
• Why did you carry out the study?
• What was the exact topic under investigation?
• Which factors and variables were included? And state why specific variables were omitted from the research scope.
• Who or what did the study explore? What was the target population?
• What was the study's location (geographical area) or setting (e.g., laboratory)?
• What was the timeframe within which you collected your data ?
• Consider a study exploring the differences between identical twins who were raised together versus identical twins who weren't. The data collection might span 5, 10, or more years.
• A study exploring a new immigration policy will cover the period since the policy came into effect and the present moment.
• How was the research conducted (research design)?
• Experimental research, qualitative, quantitative, or mixed-methods research, literature review, etc.
• What data collection tools and analysis techniques were used? e.g., If you chose quantitative methods, which statistical analysis techniques and software did you use?
• What did you find?
• What did you conclude?

Useful vocabulary for scope and delimitations

When explaining both the scope and delimitations, it's important to use the proper language to clearly state each.

For the scope , use the following language:

• This study focuses on/considers/investigates/covers the following:
• This study aims to . . . / Here, we aim to show . . . / In this study, we . . .
• The overall objective of the research is . . . / Our objective is to . . .

When stating the delimitations, use the following language:

• This [ . . . ] will not be the focus, for it has been frequently and exhaustively discusses in earlier studies.
• To review the [ . . . ] is a task that lies outside the scope of this study.
• The following [ . . . ] has been excluded from this study . . .
• This study does not provide a complete literature review of [ . . . ]. Instead, it draws on selected pertinent studies [ . . . ]

Analysis of a published scope

In one example, Simione and Gnagnarella (2020) compared the psychological and behavioral impact of COVID-19 on Italy's health workers and general population.

Here's a breakdown of the study's scope into smaller chunks and discussion of what works and why.

Also notable is that this study's delimitations include references to:

• Recruitment of participants: Convenience sampling
• Demographic characteristics of study participants: Age, sex, etc.
• Measurements methods: E.g., the death anxiety scale of the Existential Concerns Questionnaire (ECQ; van Bruggen et al., 2017) etc.
• Data analysis tool: The statistical software R

Analysis of published scope and delimitations

Scope of the study : Johnsson et al. (2019) explored the effect of in-hospital physiotherapy on postoperative physical capacity, physical activity, and lung function in patients who underwent lung cancer surgery.

The delimitations narrowed down the scope as follows:

Refine your scope, delimitations, and scientific English

English ability shouldn't limit how clear and impactful your research can be. Expert AJE editors are available to assess your science and polish your academic writing. See AJE services here .

The AJE Team

See our "Privacy Policy"

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• Limitations of the Study
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The limitations of the study are those characteristics of design or methodology that impacted or influenced the interpretation of the findings from your research. Study limitations are the constraints placed on the ability to generalize from the results, to further describe applications to practice, and/or related to the utility of findings that are the result of the ways in which you initially chose to design the study or the method used to establish internal and external validity or the result of unanticipated challenges that emerged during the study.

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Theofanidis, Dimitrios and Antigoni Fountouki. "Limitations and Delimitations in the Research Process." Perioperative Nursing 7 (September-December 2018): 155-163. .

Importance of...

Always acknowledge a study's limitations. It is far better that you identify and acknowledge your study’s limitations than to have them pointed out by your professor and have your grade lowered because you appeared to have ignored them or didn't realize they existed.

Keep in mind that acknowledgment of a study's limitations is an opportunity to make suggestions for further research. If you do connect your study's limitations to suggestions for further research, be sure to explain the ways in which these unanswered questions may become more focused because of your study.

Acknowledgment of a study's limitations also provides you with opportunities to demonstrate that you have thought critically about the research problem, understood the relevant literature published about it, and correctly assessed the methods chosen for studying the problem. A key objective of the research process is not only discovering new knowledge but also to confront assumptions and explore what we don't know.

Claiming limitations is a subjective process because you must evaluate the impact of those limitations . Don't just list key weaknesses and the magnitude of a study's limitations. To do so diminishes the validity of your research because it leaves the reader wondering whether, or in what ways, limitation(s) in your study may have impacted the results and conclusions. Limitations require a critical, overall appraisal and interpretation of their impact. You should answer the question: do these problems with errors, methods, validity, etc. eventually matter and, if so, to what extent?

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com.

Descriptions of Possible Limitations

All studies have limitations . However, it is important that you restrict your discussion to limitations related to the research problem under investigation. For example, if a meta-analysis of existing literature is not a stated purpose of your research, it should not be discussed as a limitation. Do not apologize for not addressing issues that you did not promise to investigate in the introduction of your paper.

Here are examples of limitations related to methodology and the research process you may need to describe and discuss how they possibly impacted your results. Note that descriptions of limitations should be stated in the past tense because they were discovered after you completed your research.

Possible Methodological Limitations

• Sample size -- the number of the units of analysis you use in your study is dictated by the type of research problem you are investigating. Note that, if your sample size is too small, it will be difficult to find significant relationships from the data, as statistical tests normally require a larger sample size to ensure a representative distribution of the population and to be considered representative of groups of people to whom results will be generalized or transferred. Note that sample size is generally less relevant in qualitative research if explained in the context of the research problem.
• Lack of available and/or reliable data -- a lack of data or of reliable data will likely require you to limit the scope of your analysis, the size of your sample, or it can be a significant obstacle in finding a trend and a meaningful relationship. You need to not only describe these limitations but provide cogent reasons why you believe data is missing or is unreliable. However, don’t just throw up your hands in frustration; use this as an opportunity to describe a need for future research based on designing a different method for gathering data.
• Lack of prior research studies on the topic -- citing prior research studies forms the basis of your literature review and helps lay a foundation for understanding the research problem you are investigating. Depending on the currency or scope of your research topic, there may be little, if any, prior research on your topic. Before assuming this to be true, though, consult with a librarian! In cases when a librarian has confirmed that there is little or no prior research, you may be required to develop an entirely new research typology [for example, using an exploratory rather than an explanatory research design ]. Note again that discovering a limitation can serve as an important opportunity to identify new gaps in the literature and to describe the need for further research.
• Measure used to collect the data -- sometimes it is the case that, after completing your interpretation of the findings, you discover that the way in which you gathered data inhibited your ability to conduct a thorough analysis of the results. For example, you regret not including a specific question in a survey that, in retrospect, could have helped address a particular issue that emerged later in the study. Acknowledge the deficiency by stating a need for future researchers to revise the specific method for gathering data.
• Self-reported data -- whether you are relying on pre-existing data or you are conducting a qualitative research study and gathering the data yourself, self-reported data is limited by the fact that it rarely can be independently verified. In other words, you have to the accuracy of what people say, whether in interviews, focus groups, or on questionnaires, at face value. However, self-reported data can contain several potential sources of bias that you should be alert to and note as limitations. These biases become apparent if they are incongruent with data from other sources. These are: (1) selective memory [remembering or not remembering experiences or events that occurred at some point in the past]; (2) telescoping [recalling events that occurred at one time as if they occurred at another time]; (3) attribution [the act of attributing positive events and outcomes to one's own agency, but attributing negative events and outcomes to external forces]; and, (4) exaggeration [the act of representing outcomes or embellishing events as more significant than is actually suggested from other data].

Possible Limitations of the Researcher

• Access -- if your study depends on having access to people, organizations, data, or documents and, for whatever reason, access is denied or limited in some way, the reasons for this needs to be described. Also, include an explanation why being denied or limited access did not prevent you from following through on your study.
• Longitudinal effects -- unlike your professor, who can literally devote years [even a lifetime] to studying a single topic, the time available to investigate a research problem and to measure change or stability over time is constrained by the due date of your assignment. Be sure to choose a research problem that does not require an excessive amount of time to complete the literature review, apply the methodology, and gather and interpret the results. If you're unsure whether you can complete your research within the confines of the assignment's due date, talk to your professor.
• Cultural and other type of bias -- we all have biases, whether we are conscience of them or not. Bias is when a person, place, event, or thing is viewed or shown in a consistently inaccurate way. Bias is usually negative, though one can have a positive bias as well, especially if that bias reflects your reliance on research that only support your hypothesis. When proof-reading your paper, be especially critical in reviewing how you have stated a problem, selected the data to be studied, what may have been omitted, the manner in which you have ordered events, people, or places, how you have chosen to represent a person, place, or thing, to name a phenomenon, or to use possible words with a positive or negative connotation. NOTE :   If you detect bias in prior research, it must be acknowledged and you should explain what measures were taken to avoid perpetuating that bias. For example, if a previous study only used boys to examine how music education supports effective math skills, describe how your research expands the study to include girls.
• Fluency in a language -- if your research focuses , for example, on measuring the perceived value of after-school tutoring among Mexican-American ESL [English as a Second Language] students and you are not fluent in Spanish, you are limited in being able to read and interpret Spanish language research studies on the topic or to speak with these students in their primary language. This deficiency should be acknowledged.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Senunyeme, Emmanuel K. Business Research Methods. Powerpoint Presentation. Regent University of Science and Technology; ter Riet, Gerben et al. “All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies.” PLOS One 8 (November 2013): 1-6.

Structure and Writing Style

Information about the limitations of your study are generally placed either at the beginning of the discussion section of your paper so the reader knows and understands the limitations before reading the rest of your analysis of the findings, or, the limitations are outlined at the conclusion of the discussion section as an acknowledgement of the need for further study. Statements about a study's limitations should not be buried in the body [middle] of the discussion section unless a limitation is specific to something covered in that part of the paper. If this is the case, though, the limitation should be reiterated at the conclusion of the section.

If you determine that your study is seriously flawed due to important limitations , such as, an inability to acquire critical data, consider reframing it as an exploratory study intended to lay the groundwork for a more complete research study in the future. Be sure, though, to specifically explain the ways that these flaws can be successfully overcome in a new study.

But, do not use this as an excuse for not developing a thorough research paper! Review the tab in this guide for developing a research topic . If serious limitations exist, it generally indicates a likelihood that your research problem is too narrowly defined or that the issue or event under study is too recent and, thus, very little research has been written about it. If serious limitations do emerge, consult with your professor about possible ways to overcome them or how to revise your study.

When discussing the limitations of your research, be sure to:

• Describe each limitation in detailed but concise terms;
• Explain why each limitation exists;
• Provide the reasons why each limitation could not be overcome using the method(s) chosen to acquire or gather the data [cite to other studies that had similar problems when possible];
• Assess the impact of each limitation in relation to the overall findings and conclusions of your study; and,
• If appropriate, describe how these limitations could point to the need for further research.

Remember that the method you chose may be the source of a significant limitation that has emerged during your interpretation of the results [for example, you didn't interview a group of people that you later wish you had]. If this is the case, don't panic. Acknowledge it, and explain how applying a different or more robust methodology might address the research problem more effectively in a future study. A underlying goal of scholarly research is not only to show what works, but to demonstrate what doesn't work or what needs further clarification.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Ioannidis, John P.A. "Limitations are not Properly Acknowledged in the Scientific Literature." Journal of Clinical Epidemiology 60 (2007): 324-329; Pasek, Josh. Writing the Empirical Social Science Research Paper: A Guide for the Perplexed. January 24, 2012. Academia.edu; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com; What Is an Academic Paper? Institute for Writing Rhetoric. Dartmouth College; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University.

Writing Tip

Don't Inflate the Importance of Your Findings!

After all the hard work and long hours devoted to writing your research paper, it is easy to get carried away with attributing unwarranted importance to what you’ve done. We all want our academic work to be viewed as excellent and worthy of a good grade, but it is important that you understand and openly acknowledge the limitations of your study. Inflating the importance of your study's findings could be perceived by your readers as an attempt hide its flaws or encourage a biased interpretation of the results. A small measure of humility goes a long way!

Another Writing Tip

Negative Results are Not a Limitation!

Negative evidence refers to findings that unexpectedly challenge rather than support your hypothesis. If you didn't get the results you anticipated, it may mean your hypothesis was incorrect and needs to be reformulated. Or, perhaps you have stumbled onto something unexpected that warrants further study. Moreover, the absence of an effect may be very telling in many situations, particularly in experimental research designs. In any case, your results may very well be of importance to others even though they did not support your hypothesis. Do not fall into the trap of thinking that results contrary to what you expected is a limitation to your study. If you carried out the research well, they are simply your results and only require additional interpretation.

Lewis, George H. and Jonathan F. Lewis. “The Dog in the Night-Time: Negative Evidence in Social Research.” The British Journal of Sociology 31 (December 1980): 544-558.

Yet Another Writing Tip

Sample Size Limitations in Qualitative Research

Sample sizes are typically smaller in qualitative research because, as the study goes on, acquiring more data does not necessarily lead to more information. This is because one occurrence of a piece of data, or a code, is all that is necessary to ensure that it becomes part of the analysis framework. However, it remains true that sample sizes that are too small cannot adequately support claims of having achieved valid conclusions and sample sizes that are too large do not permit the deep, naturalistic, and inductive analysis that defines qualitative inquiry. Determining adequate sample size in qualitative research is ultimately a matter of judgment and experience in evaluating the quality of the information collected against the uses to which it will be applied and the particular research method and purposeful sampling strategy employed. If the sample size is found to be a limitation, it may reflect your judgment about the methodological technique chosen [e.g., single life history study versus focus group interviews] rather than the number of respondents used.

Boddy, Clive Roland. "Sample Size for Qualitative Research." Qualitative Market Research: An International Journal 19 (2016): 426-432; Huberman, A. Michael and Matthew B. Miles. "Data Management and Analysis Methods." In Handbook of Qualitative Research . Norman K. Denzin and Yvonna S. Lincoln, eds. (Thousand Oaks, CA: Sage, 1994), pp. 428-444; Blaikie, Norman. "Confounding Issues Related to Determining Sample Size in Qualitative Research." International Journal of Social Research Methodology 21 (2018): 635-641; Oppong, Steward Harrison. "The Problem of Sampling in qualitative Research." Asian Journal of Management Sciences and Education 2 (2013): 202-210.

• << Previous: 8. The Discussion
• Next: 9. The Conclusion >>
• Last Updated: May 9, 2024 11:05 AM
• URL: https://libguides.usc.edu/writingguide

• Methodology
• Open access
• Published: 23 August 2018

The prevalence of and factors associated with inclusion of non-English language studies in Campbell systematic reviews: a survey and meta-epidemiological study

• Lauge Neimann Rasmussen   ORCID: orcid.org/0000-0001-9584-2443 1 &
• Paul Montgomery 2  

Systematic Reviews volume  7 , Article number:  129 ( 2018 ) Cite this article

24k Accesses

89 Citations

21 Altmetric

Metrics details

Studies published in languages other than English are often neglected when research teams conduct systematic reviews. Literature on how to deal with non-English studies when conducting reviews have focused on the importance of including such studies, while less attention has been paid to the practical challenges of locating and assessing relevant non-English studies. We investigated the factors which might predict the inclusion of non-English studies in systematic reviews in the social sciences, to better understand how, when and why these are included/excluded.

We appraised all Campbell Collaboration systematic reviews ( n  = 123) published to July 2016, categorising each by its language inclusiveness. We sought additional information from review authors via a questionnaire and received responses concerning 47 reviews. Data were obtained for 17 factors and we explored correlations with the number of non-English studies in the reviews via statistical regression models. Additionally, we asked authors to identify factors that support or hinder the inclusion of non-English studies.

Of 123 reviews, 108 did not explicitly exclude, and of these, 17 included non-English language studies. One factor correlated with the number of included non-English studies across all models: the number of countries in which the members of the review team work ( B -value = 0.56; SE B  = 0.24; 95% CI = 0.07–1.03; p  = 0.02). This indicates that reviews which included non-English studies were more likely to be produced by international review teams.

Our survey showed a dominance of researchers from English-speaking countries (52.9%) and review teams consisting only of team members from these countries (65.9%). The most frequently mentioned challenge to including non-English studies was a lack of resources (funding and time) followed by a lack of language resources (e.g. professional translators).

Our findings may indicate a connection between the limited inclusion of non-English studies and a lack of resources, which forces review teams to rely on their limited language skills rather than the support of professional translators. If unaddressed, review teams risk ignoring key data and introduce bias in otherwise high-quality reviews. However, the validity and interpretation of our findings should be further assessed if we are to tackle the challenges of dealing with non-English studies.

Peer Review reports

Studies published in languages other than English are often neglected when research teams conduct systematic reviews. A health technology assessment of 300 randomly sampled systematic reviews published by the Cochrane Collaboration [CC], for example, found that such studies (hereafter referred to as non-English studies) were openly excluded in more than one quarter (27%) of the reviews, while more than one third of the reviews (35%) did not state explicit language criteria. In 39% of the reviews, authors explicitly searched for non-English studies, with only 15% of all reviews including any of them [ 1 ]. Within health sciences, the relevance of non-English studies is often discussed as a question of internal validity: whether non-English studies are likely to increase or decrease bias in reviews. Hence, researchers focused on the scientific necessity to include, or to justify excluding, such studies while paying less attention to the equally important practical challenges in locating and assessing relevant non-English studies.

One stream of research has debated the risks of bias of excluding non-English studies by assessing the research designs and reporting standards of non-English and English language publications. Some studies have found that English language studies have better study design standards or higher report completeness rates than non-English studies [ 2 , 3 ], while others have found no significant differences [ 4 , 5 , 6 , 7 ]. These divergent findings might be due to differences in sampling strategies and choice of indicators, as illustrated by one study which suggested that some non-English language publications scored better on some indicators for reporting standards and worse on others in comparison with English language studies [ 8 ].

Another stream of research has approached this debate by analysing how language inclusion influences effect estimates in meta-analyses [ 7 , 9 , 10 ]. A review of 50 meta-analyses found that including non-English studies influenced effect estimates in more than half of the meta-analyses: in five cases, estimates became more positive, and in 16, less positive, while the precision of the effect estimates generally decreased [ 2 ]. Egger et al. [ 4 ], looking at reports of randomised controlled trials (RCTs) conducted in German-speaking countries, found that between 1985 and 1995 authors were more likely to report their findings in English language journals when their results were statistically significant and increasingly less likely to publish in German language journals. This suggests that non-English studies are important to include to avoid bias in reviews.

The Cochrane Handbook acknowledges the risk of bias in reviews containing exclusively English language studies and somewhat vaguely recommends ‘case-by-case’ decisions concerning the inclusion of non-English studies [ 11 ]. Similarly, the methodological guidelines for Campbell Collaboration [C2] reviews warn against the risk of language bias and encourages authors not to restrict by language [ 12 ]. Other than a statement in the C2 guidelines that the removal of language restrictions in English language databases is not a sufficient substitution for searching non-English databases, neither CC nor C2 provides any practical advice to review authors on how to deal with non-English studies. The lack of concrete advice and guidelines is problematic because non-English studies have been shown to be more cumbersome for researchers to identify than English language studies. Research databases, for example, are less rigorous in their inclusion and indexing of non-English studies [ 13 , 14 , 15 ]. Searching specialised non-English language databases using search terms in the appropriate language might alleviate this problem [ 16 , 17 ], but researchers are still limited by their own language skills or their ability to pay for the services of professional translators. For these reasons, reviewers commonly report that it is costly and time-consuming to include non-English studies and use this to justify a priori exclusion [ 18 , 19 ]. Noteworthy for the present study, the role of non-English studies appears to be largely unassessed within the social sciences [ 20 ] where publication channels are more prone to publication biases [ 21 , 22 ].

In short, the debate about non-English studies in systematic reviews is not only about the internal validity of the included studies, but also the challenges involved in accessing potentially relevant studies in any language. Any strategy for addressing these issues must be based on an understanding of how, when and why non-English studies are included or excluded from reviews in practice.

This study sought to identify and explore factors that might predict the inclusion in or exclusion from systematic reviews of studies that are in languages other than English. It also sought to extend the investigation of non-English study inclusion from the health sciences to the social sciences.

The systematic reviews published by the Campbell Collaboration constitute a relevant sample for our focus on non-English studies in the social sciences. As of July 2016, Campbell had published 123 unique reviews organised within five thematic review groups: Crime and Justice, Education, International Development, Social Welfare, and Knowledge Translation and Implementation.

Campbell states that it represents the work of a diverse group of people aiming to build a ‘world-library of systematic reviews’ [ 23 ]. Like Cochrane, Campbell seeks to ensure the quality of its reviews through the enforcement of minimum standards and peer-reviewing processes [ 12 , 18 ]. Campbell’s global ambition and the institutional support it offers to review teams means that its library comprises a sample of systematic reviews with a reasonable degree of comparability. This allowed us to systematically analyse the reviews, their critical appraisal process and their success in including relevant non-English studies.

A protocol for this study was developed in advance and agreed by a panel at the University of Oxford’s Centre for Evidence-Based Intervention, in the Department of Social Policy and Intervention.

Language inclusiveness categories

We categorised Campbell reviews according to their level of inclusion of non-English studies. Reviews that excluded non-English studies with an explicit justification in the research question or research objectives were categorised as EL-justified (i.e. English language-justified ), while those that excluded non-English studies without justifications were categorised as LOE-restricted (i.e. languages other than English-restricted ). Reviews that did not explicitly exclude non-English studies were categorised as LOE-open unless they successively included non-English studies, in which case they were LOE-inclusive . Finally, reviews that did not state language criteria were assumed to be LOE-open , an assumption tested in the statistical analysis.

Data extraction

We developed a data extraction sheet mirroring the Campbell review template for our analysis [ 24 ] to collect data on the factors that might correlate with the number of included non-English studies (see Additional file  1 ). One author (LNR) conducted the data extraction and the following coding. In cases where reviews deviated from the C2 template (e.g. that by Lum et al. [ 25 ]), sections in the given C2 review that seemed likely to contain the relevant data were read and data extracted according to the pre-specified extraction sheet, but no reviews were read in full due to resource constraints.

Abstracts were assessed to determine if they included research questions that stated a geographical focus on predominantly English-speaking countries (i.e. USA, UK, Ireland, Australia and New Zealand), which could lead to categorising the review as EL-justified . The reliability of this procedure depends upon the review teams’ compliance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) standards, which advises authors to formulate research questions using the PICO format (i.e. identifying participants, interventions, comparators and outcomes) [ 26 , 27 ]. Therefore, we also extracted any information about geographical limitations that were stated in the research objectives, as a way to identify when reviews were EL-justified .

Data were collected on the following: characteristics of the review team (number of authors; author institutions; number of different author working countries) and the systematic review (Campbell review group; publication year; inclusion criteria). In cases where a Campbell review was co-registered with the Cochrane Collaboration, this was noted, as co-registered reviews might enjoy greater institutional support during the critical-appraisal process than reviews only registered with Campbell. We also coded additional factors covering the search strategy (number of data sources sought; search terms languages; whether experts were contacted) and the flow of studies during the review process (studies located, screened, full-text assessed, included and meta-analysed). Finally, the number of non-English studies that was included in each review was estimated by counting the number of non-English titles in the list of included studies of each review.

Author questionnaire

Having extracted the majority of data from the C2 reviews, we found that some relevant factors were likely underreported to assess their importance. We therefore sent questionnaires to the review authors to inquire about factors such as the composition of their review teams (author nationalities; languages spoken) and the use of expert networks to locate studies as well as the language of applied search strings. We also asked respondents what they perceived to be the barriers and facilitators of including non-English studies. These free-text box inputs from the author questionnaire were coded iteratively and aimed to identify the challenges that review teams experience when considering, or actually including, non-English studies in Campbell reviews.

One primary author (usually the corresponding author) was invited to respond and then reminded. If needed, an invitation was sent to the entire review authorship on whom we had contact details. In cases where an author was the primary author of more than one review, a second author was prioritised. This choice was made in an effort not to overload highly productive review authors with multiple invitations. The questionnaires were completed during June and July 2016 and were followed by a consent form.

Statistical procedures

For some results, the median is the most valid estimate of the central tendencies in the dataset and are reported when appropriate.

Three exploratory multivariate models were tested with the software, SPSS Statistics 25 for Windows, to identify factors that correlate with the number of included non-English studies in the Campbell reviews. One model analysed all the LOE-open and LOE-inclusive reviews by including the 15 factors extracted from these reviews; a second per-protocol model added the questionnaire variables (author nationalities; languages spoken); and a third model was a sensitivity analysis that excluded reviews which do not explicitly state language eligibility criteria to test the robustness of our first model’s assumption that reviews are LOE-open by default. Due to the explorative nature of the study and the low statistical power from the small sample, we accepted significant associations at a p value of 0.10 when running regressions to identify possible associations.

We included all 123 unique systematic reviews published by the Campbell Collaboration and categorised each by its language inclusiveness (Table  1 ).

Based on analysis of the abstracts and research objectives of the 123 reviews, none focused solely on English-speaking countries; hence, none qualified as EL-justified . Fifteen (12.2%) reviews explicitly stated that they excluded non-English studies and are therefore categorised as LOE-restricted . The most common justifications for language exclusion were resource constraints (five cases) and lack of policy relevance outside English-speaking countries (three cases). In one case, authors mentioned that possibly relevant German and French language studies had been located but not assessed. In seven cases, no comments were made to justify the language restrictions.

Of the 123 reviews, 108 (87.8%) indicated that they were either open to studies in languages other than English ( n  = 81) or reported no language criteria ( n  = 27). Among these, we categorised 84 (68.3%) as LOE-open , 17 (13.8%) as LOE-inclusive and the remaining 7 (5.7%) as LOE-undefined , because they did not provide a list of included studies. Thirty-nine non-English studies were included in the LOE-inclusive reviews of which nine reviews contained a single non-English study, six contained two to four non-English studies, and two reviews contained respectively six and seven non-English studies. The publication languages were Spanish (13), French (11), German (5), Portuguese (5), Italian (2), Swedish (2) and Norwegian (1).

Authors of two reviews declined to participate in the questionnaire arguing that non-English studies were not relevant to the given review, or making reference to a lack of time. Responses cover 47 (38%) of the 123 reviews. Assessing differences in median figures on selected variables (Table  2 ), there is little indicating that questionnaire responders and non-responders authored substantially different systematic reviews. If valid, the answers of responders can be generalised to non-responders.

In several cases, questionnaire respondents expressed uncertainty about their co-authors’ language abilities, and some questions (those regarding the use of expert networks and language of applied search terms) prompted so vague or incomplete answers that we deemed the variables unreliable and dropped them from our analysis.

Authorship and review characteristics

We analysed language inclusiveness based on the primary subject area of each review, using the Campbell Collaboration Review Groups as indicators of subject area (Table  3 ). Eleven of the 15 reviews that excluded non-English studies without justifications (i.e. LOE-restricted ) were in Crime and Justice. Reviews that fell under the purview of the Social Welfare group represented almost half of the reviews in our study ( n  = 60), but only around 8.3% ( n  = 5) included non-English studies. International Development contained more than half of the included non-English studies ( n  = 21), although the group represents a minority (around one fifth or n  = 25) of the total reviews. In Education, there were almost as many reviews that included non-English studies ( n  = 18) as in International Development, but the former represents a slightly larger proportion of the total reviews ( n  = 29).

Thirty-nine (31.7%) C2 reviews were also registered with the CC, but only two of the co-registered reviews included non-English studies. Thus, 15 of 17 (88.2%) LOE-inclusive reviews were published exclusively by the Campbell Collaboration and accounted for 37 of the 39 (94.9%) non-English studies included in the total sample of C2 reviews.

Each review involved between four and five authors, who tended to be affiliated with two or three different institutions working within the same country (Table  4 ). Based on results of the author questionnaire, the review teams usually represented one or two nationalities and one or two languages, although teams that conducted LOE-inclusive reviews tended to speak four languages. However, in several cases, questionnaire respondents expressed some uncertainty about their co-authors’ language skills.

Reviews that included non-English studies were more likely to accept quasi-experimental designs (70.6%) in addition to RCTs, while reviews that excluded non-English studies without justifications ( LOE-restricted ) and those that did not explicitly exclude non-English studies ( LOE-open ) were less likely to accept quasi-experimental designs (20% and 57.1%, respectively).

The authors of reviews that included non-English studies were more likely to contact experts to identify relevant studies (82.4%), compared to the authors of reviews in the other two categories ( LOE-restricted  = 60%; LOE-open  = 75%).

Review teams typically searched between 21 and 26 databases, registers and journals, but rarely with non-English search terms. Only in 11 reviews did authors apply search terms in Spanish (8), Swedish (7), Portuguese (4), French (3), Norwegian (3), Danish (2) and Arabic, Chinese, Dutch, German, Italian and Russian (1 each).

Figure  1 outlines the pooled flow of studies following the PRISMA-diagram framework [ 26 ]. Each diagram represents one language category and lists the mean, median and total number of studies located, screened, full-text assessed, included and meta-analysed throughout the review process.

Synthesised study flow diagrams based on the sample of systematic reviews published by the Campbell Collaboration, excluding those seven reviews that did not provide a list of included studies [ 28 ]

The figure reveals that reviews that excluded non-English studies without justifications ( LOE-restricted ) located (median = 5151) and screened (median = 1780) substantially fewer studies than reviews that did not explicitly exclude non-English studies ( LOE-open : respective medians = 8795; 6149) and reviews that ultimately included non-English studies ( LOE-inclusive : respective medians = 9995; 4591). Reviews that included non-English studies ( LOE-inclusive ) assessed more full text studies (median = 150) than reviews that did not exclude non-English studies ( LOE-open : median = 95) though assessing less than those that excluded non-English studies ( LOE-restricted : median = 195). Noticeably, the reviews that included non-English studies also included at least twice as many studies (median = 40) than the other two categories ( LOE-open : median = 13; LOE-restricted : median = 20). However, the former review category tended to discard far more studies from meta-analysis compared to the other categories. The assessment of the LOE-inclusive reviews showed that 31 of the 39 non-English studies that were included in Campbell reviews were subsequently also included in meta-analyses.

Overall, the success rate from screening to including studies were 1.1%, 0.2% and 0.9% (based on medians in Fig.  1 ) in LOE-restricted, LOE-open and LOE-inclusive respectively—and substantially lower if one calculates success rate based on the number of located studies or the number of included non-English studies for LOE-inclusive reviews.

Regression analyses

The three exploratory regression models explain between 0.37 and 0.59 of the variation in the data (Table  5 ). In the first model ( p  = 0.05), with data from the survey of included reviews, one factor ( number of different working countries represented by authors ) was significant with a B -value of 0.56 (SE B  = 0.24; 95% CI = 0.07–1.03; p  = 0.02). This suggests that when a review team included an author working in a different country than the rest of the authorship, the review was, on average, likely to include 0.56 more non-English studies. Similar, but slightly stronger, correlations between the number of different working countries and number of included non-English studies were identified in the two other models: model 2 ( p  = 0.07) with the survey data identifies a B -value of 0.96 (SE B  = 0.43; 95% CI = 0.07–1.86; p  = 0.04) and the third model ( p  = 0.09), which excluded those reviews that did not state explicit language criteria, identifies a B -value of 0.65 (SE B  = 0.30; 95% CI = 0.05–1.25; p  = 0.04).

Two other variables, number of included studies (model 1 and model 3) and number of screened studies (model 3), showed significant correlations with the inclusion of non-English studies. The B -values for the number of included studies range between 0.01 and 0.02 indicating that including 50–100 additional studies, on average, correlates with the inclusion of an additional non-English study. The correlation between the number of screened studies and number of included non-English studies in model 3 is, though significant, substantially un-interpretable at first ( B -value = 0.00; SE B  = 0.00; 95% CI = 0.00–0.00; p  = 0.09). However, as standardised coefficients (beta), their magnitude (model 1: included studies = 0.34; model 3: included studies = 0.40, screened studies = − 0.55) are equivalent to the standardised coefficients of the number of author countries ranging between 0.41 and 0.48. Substantially, this could be interpreted as an indication that the more studies review teams include, the more non-English studies they are likely to include, while the more studies review teams screen, the less likely they are to include non-English studies.

To counter non-normal data distribution, all models were bootstrapped with 1000 samples. None of the models were significant after this procedure. Countering substantial multicollinearity, a simpler model with seven variables (number of authors, author institutions, author working countries, methodological criteria, sources searched, use of experts and search-term languages) was tested and again identified a positive relationship between the number of working countries represented by authors and the number of reviews that included non-English studies.

Assessing ‘author country’ more closely, we found that 52.9% of review authors worked in the USA or the UK (Table  6 ). In fact, 65.9% of the teams ( n  = 81) only had members working in English-speaking countries, while 34.1% ( n  = 42) had one or more members working outside an English-speaking country.

Barriers to and facilitators of including non-English studies

Unsurprisingly, authors commonly pointed to issues of cost, time and funding as crucial for the inclusion of non-English studies, as well as lack of language resources (Table  7 ). ‘Language resources’ here refers to people or services external to the review team (e.g. professional and volunteer translators, software translation tools and English abstracts). ‘Language skills’—the language competencies within the review teams (e.g. multilingual authors and affiliated staff)—was not experienced as a barrier, nor a facilitator, as often as language resources, but was still pointed to as the third most common barrier. Slightly more often than language skills, authors pointed to the need for training in and guidelines on how to deal with non-English studies and access to non-English specialised databases as important facilitators. Issues of bias and methodological quality were mentioned, although infrequently.

Among the 123 reviews in our study, 108 did not exclude non-English studies a priori, and of those who did, few justified their reasons to do so . The relatively low prevalence of non-English studies in our sample of reviews might be somewhat underestimated by our data extraction approach, counting non-English titles in the study inclusion list. Assuming that this is not the case, the low prevalence might indicate that relevant non-English studies were not available or that the review teams failed to identify these studies. We did not assess whether relevant non-English studies had been overlooked or, if located, were excluded due to risk of bias. Overall, however, the infrequent number of non-English studies does leave some room for C2 to convincingly develop a ‘world-library of systematic reviews’ [ 23 ].

The higher acceptance of quasi-experimental designs by reviews that included non-English studies might be interpreted as an indication of lower methodological criteria thresholds. However, the relevance of studies does not depend simply on their position in the hierarchy of evidence but also on other factors such as the rigour by which they have been conducted and the contextual feasibility of research designs for a given research topic. We assumed that, by following the Cochrane Collaboration standards, the reviews published by Campbell included rigorous critical appraisal of all included studies. With this assumption, our statistical analyses did not indicate that the methodological threshold or any other step of the critical appraisal process affected whether non-English studies were included. We also did not find any indications that co-registered reviews with institutional support from both Cochrane and Campbell were more likely to include non-English studies than those published exclusively by Campbell.

Results of the author questionnaire suggested that the most obvious challenges to include non-English studies were resource constraints and, somewhat linked to this, the reliance of research teams on their own internal language skills. In this light, Fig.  1 illustrates that review teams may expect an overwhelming number of studies to screen and full-text assess when seeking to include non-English studies. To counter this challenge, we suggest two options that could lower the work load burden for C2 review teams and improve the review quality. First, teams might benefit from putting more effort into improving the specificity of their research questions and search strategies. Our regression analyses (Table  5 ) indicated a negative relationship between the number of studies screened and the inclusion of non-English studies, as well as a positive relationship between the number of studies included and the inclusion of non-English studies. These correlations were not consistent between our regression models; thus, the results and interpretations are somehow speculative but could suggest that authors of LOE-inclusive reviews conducted searches that more successfully than authors of LOE-open reviews balanced sensitivity and specificity. Indeed, Fig.  1 does illustrate that LOE-inclusive reviews succeed in including more relevant studies disregarding publication language than LOE-open reviews, while screening substantially fewer studies.

Second, more review teams could consider explicitly restricting their reviews to English language publications and state, as well as justify, this limitation, e.g. in abstracts, research questions, objectives and eligibility criteria. Pragmatically restricting reviews to English publications is legitimate but should be clearly acknowledged and the limitations in findings and their relevance should then be properly discussed by review teams. Future C2 guidelines could address these issues more clearly as called for by some of our questionnaire respondents.

Considering that our statistical models indicated that the composition of review teams working across countries significantly correlates with the number of included non-English studies, one can speculate whether more international review teams master more languages than less international teams and that this perhaps allows the former to pursue the identification of non-English studies more diligently. This speculation is not supported by our statistical models, which did not identify language as being of significant importance. However, the data on author languages was to some degree unreliable as questionnaire respondents expressed uncertainty about languages spoken by their co-authors. Further, the questionnaire only covered 47 of the 123 review teams, which lowers its statistical power to identify a real relationship, if one exists. The statistical power of another language variable identified by earlier research [ 16 , 17 ]—the application of non-English search term in the literature search process—is also low due to the few reviews that applied non-English search terms. We therefore cannot confirm the importance of language in accessing non-English studies, nor do we have reason to reject the importance of language diversity.

An alternative interpretation of the statistical relationship between author countries and included non-English studies is that international review teams have easier, perhaps informal, access to a more diverse set of language resources than teams working within the same country. It might also be that the range of author countries is a proxy for knowledge about and access to more diverse or specific publication channels that facilitate the inclusion of non-English studies. Finally, there might be a degree of selection effect operating, whereby international review teams pick research topics with more global relevance and therefore a higher prevalence of non-English studies.

Limitations

A main limitation of the present study relates to its exploratory nature and the statistical robustness of the findings. First, only one author (LNR) conducted the data extraction and coding, meaning there could be a degree of bias and risk of errors in the process. However, a protocol was put in place to guide the project and frequent support and supervision was given with the second author. Some caution is also warranted considering the statistical issues of non-normality, relatively high levels of multicollinearity and chances of random error when dealing with 15 to 17 factors within a relatively small dataset. Still, the relation between author countries and included non-English studies was consistent for all models, except for the bootstrapped ones, which added some credibility to the results, supported by the qualitative data. Unfortunately, the results are somewhat confounded by the 35 systematic reviews—accounting for 17.9% of the total Campbell authorship (Table  6 )—that did not report exhaustively on the institutional affiliation of all review authors. Some statistical power could perhaps have been gained had we had the resources to read the 123 reviews in full, e.g. in the hope of identifying the publication languages of those individual studies that were included in the seven C2 reviews which, surprisingly, did not provide a basic list of included studies.

There are also limits to the depth of the dataset. We found few and smaller differences when comparing the three review categories (Table  4 ), for example in relation to the number of data sources sought. In practice, however, the number of data sources might be less relevant than which (non-English language specialised) data sources a review team searched. Perhaps the clear dominance of individual researchers based in English-speaking countries and review teams consisting only of team members in these countries reflects a partiality among publication channels for studies in English. Working country is not synonymous with the origin of authors and thereby which languages review teams might master, but it is possible that our survey did not yield sufficient, nor adequately reliable, information to identify a possible association with the number of included non-English studies. Additionally, if the low prevalence of non-English search terms is a proxy for the general rigour with which non-English studies have been pursued, the factor that we identified (authors’ working countries) might not be the most effective. Factors such as the number of search-term languages might be more important in practice if they were applied more often.

At the moment, we cannot tell to which degree the results can be extrapolated from our sample of Campbell reviews to the wider population of reviews. Knowing the differences in publication channels between social sciences and health sciences [ 21 ], and considering the substantial differences in including non-English studies between the reviews in our sample that were co-published with the CC and those published exclusively by the C2, we would encourage the replication of this study’s research design, for example with a sample of systematic reviews from the Cochrane Collaboration.

Finally, we believe new perspectives and a deeper understanding of the systematic challenges in dealing with non-English studies could be obtained by approaching the issue through more qualitative methods. Interviews with internationally experienced reviewers could, for example help map out more extensively the practical barriers and facilitators for the inclusion of non-English studies in systematic reviews. To our knowledge, such a study design would be the first of its kind on an issue that has been dominated by quantitative study designs.

We investigated the factors that might predict the inclusion of studies that are in languages other than English in systematic reviews, particularly in the social sciences. We analysed all 123 systematic reviews published by the Campbell Collaboration, categorising each by its language inclusiveness. We also sought additional data from review authors and received responses from around one third of our sample.

The majority of Campbell reviews ( n  = 108) did not explicitly exclude non-English language studies, and 17 (13.8%) actually included non-English language studies. The most obvious challenge to including non-English studies, according to review authors, was cost and time. This might be a key reason for another common obstacle: review teams’ reliance only on their own language skills, rather than calling on the support of professional translators.

Overall, our sample of reviews showed a clear dominance of individual researchers based in English-speaking countries and review teams consisting only of team members in these countries, which could reflect a partiality among social science publication channels for studies in English.

Reviews which included non-English studies were more likely to be produced by review teams comprised of members working across different countries and languages. However, the reasons for this are unclear. For example, international review teams may have easier, perhaps informal, access to and/or knowledge about a more diverse set of language resources and publication channels than teams working within the same country. Or there might be a degree of selection effect in play, whereby international review teams pick research topics with more global relevance and therefore a higher prevalence of non-English studies.

This study has highlighted some of the remaining questions around language inclusiveness in systematic reviews and the unique challenges involved in locating and assessing available non-English studies. These studies might ensure the internal validity of findings, or perhaps increase external validity to the degree that reviews with non-English studies differ from those with only English language studies with respect to the location, culture and specific population groups they represent. In light of these issues, we recommend replicating our study using a wider range of reviews, for example using the Cochrane Library as a sample. Such efforts are crucial if the evidence-based movement is to succeed in becoming a global movement of people aiming to build a world library of systematic reviews.

Abbreviations

English language-justified, i.e. reviews that exclude non-English studies with an explicit justification in the research question or research objectives

Languages other than English-inclusive, i.e. reviews that include non-English studies

Languages other than English-open, i.e. reviews that do not explicitly exclude non-English studies

Languages other than English-restricted, i.e. reviews that explicitly exclude non-English studies without justifications

Languages other than English-undefined, i.e. reviews that do not provide a list of included studies

Preferred Reporting Items for Systematic Reviews and Meta-analyses

Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14:8.

Article   CAS   Google Scholar  

Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol. 2002;31(1):115–23.

Article   PubMed   Google Scholar  

Zhang D, Freemantle N, Cheng KK. Are randomized trials conducted in China or India biased? A comparative empirical analysis. J Clin Epidemiol. 2011;64(1):90–5.

Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet. 1997;350(9074):326–9.

Article   PubMed   CAS   Google Scholar  

Klassen TP, Pham B, Lawson ML, Moher D. For randomized controlled trials, the quality of reports of complementary and alternative medicine was as good as reports of conventional medicine. J Clin Epidemiol. 2005;58(8):763–8.

Moher D, Fortin P, Jadad AR, Jüni P, Klassen T, Le Lorier J, Liberati A, Linde K, Penna A. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet. 1996;347:363–6.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess. 2003;7:41.

Article   Google Scholar  

Shiwa SR, Moseley AM, Maher CG, Pena Cosa LO. Language of publication has a small influence on the quality of reports of controlled trials of physiotherapy interventions. J Clin Epidemiol. 2013;66(1):78–84.

Grégoire G, Derderian F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol. 1995;48(1):159–63.

Moher D, Klassen TP, Schulz KF, Berlin JA, Jadad AR, Liberati A. What contributions do languages other than English make on the results of meta-analyses? J Clin Epidemiol. 2000;53(9):964–72.

Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration;sec. 10.2.2.4. 2011 http://handbook.cochrane.org/front_page.htm . Accessed 30 Jan 2017.

Google Scholar  

Chandler Churchill, Higgins, Lasserson, Tovey. Methodological Expectations of Campbell Collaboration Intervention Reviews (MEC2IR): DRAFT Version 0.1, June 23, 2014 (Updated September 2014). Adaptations on MECIR Version 2.2 Conduct Standards. The Campbell Collaboration. 2014. http://archive.campbellcollaboration.org/artman2/uploads/1/MEC2IR_reporting_standards_v1_0_Updated_September_2014_1.pdf . Accessed 30 Jan 2017.

Loria A, Arroyo P. Language and country preponderance trends in MEDLINE and its causes. J Med Libr Assoc. 2005;93(3):381–5.

PubMed   PubMed Central   Google Scholar  

Wu XY, Tang JL, Chen M, Yuan JQ, Qin Y, Chung VCH. Systematic reviews and meta-analyses of traditional Chinese medicine must search Chinese databases to reduce language bias. Evid Based Complement Alternat Med. 2013;2013:1-7.

Türp JC, Schulte JM, Antes G. Nearly half of dental randomized controlled trials published in German are not included in Medline. Eur J Oral Sci. 2002;110(6):405–11.

Cohen JF, Korevaar DA, Wang J, Spijker R, Bossuyt PM. Should we search Chinese biomedical databases when performing systematic reviews? Syst Rev. 2015;4:23.

Article   PubMed   PubMed Central   Google Scholar  

Xia J, Wright J, Adams CE. Five large Chinese biomedical bibliographic databases: accessibility and coverage. Health Inf Libr J. 2008;25(1):55–61.

Hammerstrøm K, Wade A, Klint Jørgensen AM. Searching for studies: a guide to information retrieval for Campbell systematic reviews. The Campbell Collaboration. 2010. http://www.campbellcollaboration.org/images/pdf/plain-language/Hammerstrom_Guide_IR_Supplement.pdf . Accessed 30 Jan 2017.

Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7:1.

PubMed   CAS   Google Scholar  

Shenderovich Y, Eisner M, Mikton C, Gardner F, Liu J, Murray J. Methods for conducting systematic reviews of risk factors in low- and middle-income countries. BMC Med Res Methodol. 2016;16:32.

Grayson L, Gomersall A. A difficult business: finding the evidence for social science reviews. In: ESRC UK Centre for Evidence Based Policy and Practice. Working Paper 19; 2003. https://www.kcl.ac.uk/sspp/departments/politicaleconomy/research/cep/pubs/papers/assets/wp19.pdf . Accessed 30 Jan 2017.

Hopewell S, Clarke M, Mallett S. Grey literature and systematic reviews’. In: Rothstein HR, Sutton AJ, Borenstein M, editors. Publication bias in meta-analysis: prevention, assessment and adjustment. West Sussex: John Wiley & Sons, Ltd; 2005. p. 49–72.

Steering Group of Campbell Collaboration. A strategic plan for the Campbell collaboration. The Campbell collaboration. 2008. https://www.campbellcollaboration.org/images/pdf/plain-language/Strategic_plan_C2_vers_1.0_17_Oct_08.pdf . Accessed 30 Jan 2017.

The Campbell Collaboration. Producing a review: The Campbell collaboration. 2016. https://campbellcollaboration.org/library/campbell-systematic-review-templates.html . Accessed 22 Feb 2016.

Lum C, Kennedy L, Sherley A. The effectiveness of counter-terrorism strategies. Campbell systematic reviews; 2006. p. 2.

Book   Google Scholar  

Moher D, Liberati A, Tetzlaff J, Altman DG, The Prisma Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–9.

Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, Gøtzsche PC, Lasserson T, Tovey D, for the PRISMA for Abstracts Group. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013;10(4):e1001419.

Simpson S, Rorie M, Alper ME, Schell-Busey N, Laufer W, Smith NC. Corporate crime deterrence: a systematic review. Campbell systematic reviews, vol. 10; 2014. p. 4.

Download references

Acknowledgements

The authors acknowledge Julia Littell who gave valuable advice and feedback on the manuscript and would also like to thank the questionnaire respondents for their help with the survey. Finally, we would like to acknowledge the reviewers whose constructive critique improved the article substantially.

Availability of data and materials

All data analysed during the study are available from the corresponding author on reasonable request, but restrictions apply to the availability of parts of the questionnaire data due to privacy of respondents.

Author information

Authors and affiliations.

The Danish Medicines Council, Copenhagen, Denmark

Lauge Neimann Rasmussen

Department of Social Policy, Sociology and Criminology, University of Birmingham, Birmingham, UK

Paul Montgomery

You can also search for this author in PubMed   Google Scholar

Contributions

LNR conceptualised the study with PM contributing substantially to the design of the study as well as the acquisition, analysis and interpretation of the data. LNR extracted and coded the data and drafted the manuscript with substantial revision by PM. Both authors have given final approval of the version to be published.

Corresponding author

Correspondence to Lauge Neimann Rasmussen .

Ethics declarations

Ethics approval and consent to participate.

This study was originally conducted as a MSc thesis as part of the MSc programme in Evidence-Based Social Interventions and Policy Evaluation at the University of Oxford’s Centre for Evidence-Based Intervention, in the Department of Social Policy and Intervention. A protocol for the study was developed in advance and agreed by a panel at the Centre for Evidence-Based Intervention.

Consent for publication

Written consent for publication was received from all questionnaire respondents.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

File containing the data extraction sheet used to collect data on the Campbell Collaboration systematic reviews. (PDF 498 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Neimann Rasmussen, L., Montgomery, P. The prevalence of and factors associated with inclusion of non-English language studies in Campbell systematic reviews: a survey and meta-epidemiological study. Syst Rev 7 , 129 (2018). https://doi.org/10.1186/s13643-018-0786-6

Download citation

Received : 16 March 2017

Accepted : 27 July 2018

Published : 23 August 2018

DOI : https://doi.org/10.1186/s13643-018-0786-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic reviews
• Language bias
• Location bias
• Country bias
• Campbell Collaboration

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Challenges and Limitations of Geospatial Data and Analyses in the Context of COVID-19

• First Online: 15 July 2021

Cite this chapter

• Sean G. Young 5 ,
• Jyotishka Datta 6 ,
• Bandana Kar 7 ,
• Xiao Huang 8 ,
• Malcolm D. Williamson 9 ,
• Jason A. Tullis 8 &
• Jackson Cothren 8  

Part of the book series: Human Dynamics in Smart Cities ((HDSC))

598 Accesses

4 Citations

The global COVID-19 pandemic has infected public consciousness with a new appreciation of geospatial analyses in public health. Not since John Snow’s 1854 cholera map of London has a public health crisis brought so much attention to geospatial data, and their importance to understanding, predicting, and preventing the spread of disease. Although high profile models and visualization dashboards of COVID-19 have increased awareness of geospatial data and analytics in public health, they have also amplified concerns regarding data issues, including questions of reproducibility and replicability (R&R), provenance, scale of analysis, fitness for use, trust, privacy, and uncertainty. The following chapter sections discuss these data issues and their implications for decision making during the COVID-19 pandemic. R&R and provenance —Many complex geospatial R&R issues can be simplified by a focus on who has (or should have) access to associated geospatial provenance information which is here defined. Human mobility data —Non-traditional data sources, such as social media, can augment official reporting sources and provide insight into mobility and social distancing behaviors. We discuss how to incorporate human mobility data selectively, while maintaining anonymity, with official data sources during emergency situations. Scales of analysis —Although several models and geo-visualization tools are providing information about disease spread based on mobility, network connectivity, and/or socio-economic conditions, these models and tools rarely shed light about the impact of the spatial scale at which data are available and models are implemented. The widely adopted practice in the U.S. of mapping data to counties may provide false impressions to policy makers and the public regarding the locations at greatest and least risk. Given the dynamic nature of the disease, temporal scale is also crucial to understanding the implications of models. Understanding uncertainty in projection models —A key issue in infectious disease epidemiological models is the characterization and quantification of uncertainty in predictions. Uncertainties come from assumptions on model parameters, model estimation, and both systematic and random uncertainties from input data, including temporal lags in reporting. Model choice may add to overall uncertainties in the predicted numbers. Since these forecasts have an immense impact on public health policy and individual risk-based decision making, it is paramount to understand and convey the sources and nature of uncertainties.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Atkinson, P. M., & Tate, N. J. (2000). Spatial scale problems and geostatistical solutions: A review. The Professional Geographer, 52, 607–623. https://doi.org/10.1111/0033-0124.00250 .

Article   Google Scholar  

Barrett, F. A. (2000). Disease & geography: The history of an idea . Atkinson College, Dept. of Geography, Toronto.

Google Scholar  

Begley, S. (2020). Influential COVID-19 models shouldn’t guide U.S. policies, critics say. In: Statnews.com. https://www.statnews.com/2020/04/17/influential-covid-19-model-uses-flawed-methods-shouldnt-guide-policies-critics-say/ . Accessed 20 Aug 2020.

Bertozzi, A. L., Franco, E., Mohler, G., Short, M. B., & Sledge, D. (2020). The challenges of modeling and forecasting the spread of COVID-19. ArXiv200404741 Q-Bio.

Boccara, N., & Cheong, K. (1992). Automata network SIR models for the spread of infectious diseases in populations of moving individuals. Journal of Physics A: Mathematical and General, 25, 2447–2461. https://doi.org/10.1088/0305-4470/25/9/018 .

Booker, B. (2020). Japan to declare nationwide state of emergency as virus spreads. In Japan Times. https://www.japantimes.co.jp/news/2020/04/16/national/japan-nationwide-state-of-emergency-coronavirus/ . Accessed 20 Aug 2020.

Diekmann, O., Heesterbeek, J. A. P., & Roberts, M. G. (2010). The construction of next-generation matrices for compartmental epidemic models. Journal of the Royal Society, Interface, 7, 873–885. https://doi.org/10.1098/rsif.2009.0386 .

Estes, L., Elsen, P. R., Treuer, T., Ahmed, L., Caylor, K., Chang, J., et al. (2018). The spatial and temporal domains of modern ecology. Nature Ecology and Evolution, 2, 819–826. https://doi.org/10.1038/s41559-018-0524-4 .

Fisher, D., & Wilder-Smith, A. (2020). The global community needs to swiftly ramp up the response to contain COVID-19. The Lancet, 395, 1109–1110. https://doi.org/10.1016/S0140-6736(20)30679-6 .

Friedman, J., Liu, P., Gakidou, E., & IHME COVID-19 Model Comparison Team. (2020). Predictive performance of international COVID-19 mortality forecasting models. Epidemiology .

Gelernter, J., & Mushegian, N. (2011). Geo-parsing messages from microtext. Transactions in GIS, 15, 753–773. https://doi.org/10.1111/j.1467-9671.2011.01294.x .

Goodchild, M. F. (2007). Citizens as sensors: The world of volunteered geography. GeoJournal, 69, 211–221. https://doi.org/10.1007/s10708-007-9111-y .

Goodchild, M. F., & Quattrochi, D. A. (1997). Scale, multiscaling, remote sensing, and GIS. In M. F. Goodchild & D. A. Quattrochi (Eds.), Scale in remote sensing and GIS (pp. 1–12). Florida, USA: CRC Press.

Hamidi, S., Sabouri, S., & Ewing, R. (2020). Does density aggravate the COVID-19 pandemic?: Early findings and lessons for planners. Journal of the American Planning Association, 86, 495–509. https://doi.org/10.1080/01944363.2020.1777891 .

Hawelka, B., Sitko, I., Beinat, E., Sobolevsky, S., Kazakopoulos, P., & Ratti, C. (2014). Geo-located Twitter as proxy for global mobility patterns. Cartography and Geographic Information Systems, 41, 260–271. https://doi.org/10.1080/15230406.2014.890072 .

Hippocrates. (1849). The genuine works of Hippocrates . New York: Printed for the Sydenham society.

Huang, X., Li, Z., Jiang, Y., Li, X., & Porter, D. (2020a). Twitter reveals human mobility dynamics during the COVID-19 pandemic. PLoS ONE, 15, . https://doi.org/10.1371/journal.pone.0241957 .

Huang, X., Li, Z., Jiang, Y., Ye, X., Deng, C., Zhang, J., & Li, X. (2020b). The characteristics of multi-source mobility datasets and how they reveal the luxury nature of social distancing in the U.S. during the COVID-19 pandemic. medRxiv 2020.07.31.20143016. https://doi.org/10.1101/2020.07.31.20143016 .

Huang, X., Li, Z., Wang, C., & Ning, H. (2020c). Identifying disaster related social media for rapid response: A visual-textual fused CNN architecture. International Journal of Digital Earth, 13, 1017–1039. https://doi.org/10.1080/17538947.2019.1633425 .

Ivorra, B., Ferrández, M. R., Vela-Pérez, M., & Ramos, A. M. (2020). Mathematical modeling of the spread of the coronavirus disease 2019 (COVID-19) taking into account the undetected infections. The case of China. Communications in Nonlinear Science and Numerical Simulation, 88 , 105303. https://doi.org/10.1016/j.cnsns.2020.105303 .

Jewell, N. P., Lewnard, J. A., & Jewell, B. L. (2020). Caution warranted: Using the Institute for Health Metrics and Evaluation model for predicting the course of the COVID-19 pandemic. Annals of Internal Medicine, 173, 226–227. https://doi.org/10.7326/M20-1565 .

Jun, S. W. (2020). Movement control order not a lockdown, says former health minister. In: Malay Mail. https://www.malaymail.com/news/malaysia/2020/03/17/movement-control-order-not-a-lockdown-says-former-health-minister/1847232 . Accessed 20 Aug 2020.

Jurdak, R., Zhao, K., Liu, J., AbouJaoude, M., Cameron, M., & Newth, D. (2015). Understanding human mobility from Twitter. PLoS ONE, 10, . https://doi.org/10.1371/journal.pone.0131469 .

Kar, B. (2008). Scaling modeled potential residential loss from a storm surge . Ph.D., University of South Carolina.

Kar, B., & Hodgson, M. E. (2012). Observational scale and modeled potential residential loss from a storm surge. GIScience & Remote Sensing, 49, 202–227. https://doi.org/10.2747/1548-1603.49.2.202 .

Karaye, I. M., & Horney, J. A. (2020). The impact of social vulnerability on COVID-19 in the U.S.: An analysis of spatially varying relationships. American Journal of Preventive Medicine, 59, 317–325. https://doi.org/10.1016/j.amepre.2020.06.006 .

Kedron, P., Frazier, A. E., Trgovac, A. B., Nelson, T., & Fotheringham, A. S. (2019). Reproducibility and replicability in geographical analysis. Geographical Analysis . https://doi.org/10.1111/gean.12221 .

Kermack, W. O., & McKendrick, A. G. (1927). A contribution to the mathematical theory of epidemics. Proceedings of the Royal Society London, 115, 700–721.

Kraemer, M. U. G., Yang, C.-H., Gutierrez, B., Wu, C.-H., Klein, B., Pigott, D. M., Covid, O., Hanage, W. P., Brownstein, J. S., Layan, M., Vespignani, A., Tian, H., Dye, C., Pybus, O. G., & Scarpino, S. V. (2020). The effect of human mobility and control measures on the COVID-19 epidemic in China. 6.

Kuniya, T. (2020). Prediction of the epidemic peak of coronavirus disease in Japan, 2020. Journal of Clinical Medicine, 9, 789. https://doi.org/10.3390/jcm9030789 .

Lam, N. S.-N., & Quattrochi, D. A. (1992). On the issues of scale, resolution, and fractal analysis in the mapping sciences*. The Professional Geographer, 44, 88–98. https://doi.org/10.1111/j.0033-0124.1992.00088.x .

Leguay, J., Friedman, T., & Conan, V. (2006). Evaluating mobility pattern space routing for DTNs. In Proceedings of the IEEE INFOCOM 2006 25th IEEE international conference on computer communications (pp. 1–10). https://doi.org/10.1109/INFOCOM.2006.299 .

Lekone, P. E., & Finkenstädt, B. F. (2006). Statistical inference in a stochastic epidemic SEIR model with control intervention: Ebola as a case study. Biometrics, 62, 1170–1177. https://doi.org/10.1111/j.1541-0420.2006.00609.x .

Li, Z., Wang, C., Emrich, C. T., & Guo, D. (2018). A novel approach to leveraging social media for rapid flood mapping: a case study of the 2015 South Carolina floods. Cartography and Geographic Information Systems, 45, 97–110. https://doi.org/10.1080/15230406.2016.1271356 .

Mandelbrot, B. (1967). How long is the coast of Britain? Statistical self-similarity and fractional dimension. Science, 156, 636–638. https://doi.org/10.1126/science.156.3775.636 .

Marchant, R., Samia, N. I., Rosen, O., Tanner, M. A., & Cripps, S. (2020). Learning as we go: An examination of the statistical accuracy of COVID19 daily death count predictions. ArXiv200404734 Quantitative Biology Status.

Martín, Y., Li, Z., & Cutter, S. L. (2017). Leveraging Twitter to gauge evacuation compliance: Spatiotemporal analysis of Hurricane Matthew. PLoS ONE, 12, . https://doi.org/10.1371/journal.pone.0181701 .

Martín, Y., Cutter, S. L., & Li, Z. (2020). Bridging twitter and survey data for evacuation assessment of Hurricane Matthew and Hurricane Irma. Natural Hazards Review, 21, 04020003. https://doi.org/10.1061/(ASCE)NH.1527-6996.0000354 .

McAfee, A., & Brynjolfsson, E. (2012) Big data: The management revolution. Harvard Business Review 60–68.

Morstatter, F., Pfeffer, J., Liu, H., & Carley, K. M. (2013). Is the sample good enough? Comparing data from Twitter’s Streaming API with Twitter’s Firehose. ArXiv13065204 Physics.

Murray CJ, IHME COVID-19 health service utilization forecasting team. (2020). Forecasting the impact of the first wave of the COVID-19 pandemic on hospital demand and deaths for the USA and European Economic Area countries. Infectious Diseases (except HIV/AIDS) .

National Academies. (2019). Reproducibility and replicability in science, of Sciences, Engineering, and Medicine . Washington, D.C.: National Academies Press.

National Institute of Standards and Technology. (2020). Research Data Framework (RDaF): Motivation, development, and a preliminary framework core .

O’Reilly, T. (2005). What is Web 2.0? Design patterns and business models for the next generation of software. In: OReilly.com. https://www.oreilly.com/pub/a/web2/archive/what-is-web-20.html . Accessed 20 Aug 2020.

Openshaw, S. (1984). The modifiable areal unit problem. Concepts and techniques in modern geography (pp. 1–41). Norwich: Geo Books.

Osthus, D., Hickmann, K. S., Caragea, P. C., Higdon, D., & Del Valle, S. Y. (2017). Forecasting seasonal influenza with a state-space SIR model. The Annals of Applied Statistics, 11, 202–224. https://doi.org/10.1214/16-AOAS1000 .

Qiu, J., Shen, B., Zhao, M., Wang, Z., Xie, B., & Xu, Y. (2020). A nationwide survey of psychological distress among Chinese people in the COVID-19 epidemic: Implications and policy recommendations. Gen Psychiatry, 33, . https://doi.org/10.1136/gpsych-2020-100213 .

Quijas, S., Balvanera, P. (2013).Biodiversity and ecosystem services. In: Encyclopedia of biodiversity (pp 341–356). Elsevier.

Ray, D., Salvatore, M., Bhattacharyya, R., Wang, L., Du, J., Mohammed, S., et al. (2020). Predictions, role of interventions and effects of a historic national lockdown in India’s response to the COVID-19 pandemic: Data science call to arms. Harvard Data Science Review . https://doi.org/10.1162/99608f92.60e08ed5 .

Roy, A., Kar, B. (2020). Characterizing the spread of COVID-19 from human mobility patterns and SocioDemographic indicators. In Proceedings of the 3rd ACM SIGSPATIAL international workshop on advances in resilient and intelligent cities (pp 39–48). ACM, Seattle Washington.

Sloan, L., & Morgan, J. (2015). Who Tweets with their location? Understanding the relationship between demographic characteristics and the use of geoservices and geotagging on Twitter. PLoS ONE, 10, . https://doi.org/10.1371/journal.pone.0142209 .

Song, P. X., Wang, L., Zhou, Y., He, J., Zhu, B., Wang, F., Tang, L., & Eisenberg, M. (2020). An epidemiological forecast model and software assessing interventions on COVID-19 epidemic in China. Infectious Diseases (except HIV/AIDS) .

Tullis, J. A., & Kar, B. (2020). Where is the provenance? Ethical replicability and reproducibility in giscience and its critical applications. American Association of Geographers Annals . https://doi.org/10.1080/24694452.2020.1806029 .

Tullis, J. A., Alsumaiti, T. S., Cothren, J. D., Lanter, D. P., Limp, W. F., Linck, R. F., Shi, X., & Young, S. G. (2015). Geoprocessing, workflows, and provenance. In P. S. Thenkabail (Ed.) Remote sensing handbook . Remotely sensed data characterization, classification, and accuracies (1st edn, vol. 1, p. 678). Taylor & Francis.

W3C Provenance Incubator Group. (2010). Provenance XG final report . W3C.

Wang, Z., & Tang, K. (2020). Combating COVID-19: Health equity matters. Nature Medicine, 26, 458. https://doi.org/10.1038/s41591-020-0823-6 .

Wang, J., Hu, Y., & Joseph, K. (2020a). NeuroTPR: A neuro-net toponym recognition model for extracting locations from social media messages. Transactions in GIS, 24, 719–735. https://doi.org/10.1111/tgis.12627 .

Wang, C., Liu, L., Hao, X., Guo, H., Wang, Q., Huang, J., He, N., Yu, H., Lin, X., Pan, A., Wei, S., & Wu, T. (2020b). Evolving epidemiology and impact of non-pharmaceutical interventions on the outbreak of coronavirus disease 2019 in Wuhan, China. Epidemiology .

Webb Hooper, M., Nápoles, A. M., & Pérez-Stable, E. J. (2020). COVID-19 and racial/ethnic disparities. JAMA, 323, 2466. https://doi.org/10.1001/jama.2020.8598 .

Williams, M. L., Burnap, P., & Sloan, L. (2017). Towards an ethical framework for publishing Twitter data in social research: Taking into account users’ views, online context and algorithmic estimation. Sociology, 51, 1149–1168. https://doi.org/10.1177/0038038517708140 .

Xu, P., Dredze, M., Broniatowski, D. A. (2020). The twitter social mobility index: Measuring social distancing practices from geolocated tweets. ArXiv200402397 Cs.

Download references

Acknowledgements

This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. The manuscript contents are solely the opinions of the authors and do not constitute a statement of policy, decision, or position on behalf of Oak Ridge National Laboratory, UT-Battelle, the Department of Energy, or the US Government.

Author information

Authors and affiliations.

Department of Environmental and Occupational Health, University of Arkansas for Medical Sciences, Little Rock, AR, USA

Sean G. Young

Department of Statistics, Virginia Tech, Blacksburg, VA, USA

Jyotishka Datta

Oak Ridge National Laboratory, Oak Ridge, TN, USA

Bandana Kar

Department of Geosciences and Center for Advanced Spatial Technologies, University of Arkansas, Fayetteville, AR, USA

Xiao Huang, Jason A. Tullis & Jackson Cothren

Center for Advanced Spatial Technologies, University of Arkansas, Fayetteville, AR, USA

Malcolm D. Williamson

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sean G. Young .

Editor information

Editors and affiliations.

Department of Geography, University of Tennessee, Knoxville, TN, USA

Shih-Lung Shaw

Geography and Public & International Affairs, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Young, S.G. et al. (2021). Challenges and Limitations of Geospatial Data and Analyses in the Context of COVID-19. In: Shaw, SL., Sui, D. (eds) Mapping COVID-19 in Space and Time. Human Dynamics in Smart Cities. Springer, Cham. https://doi.org/10.1007/978-3-030-72808-3_8

Download citation

DOI : https://doi.org/10.1007/978-3-030-72808-3_8

Published : 15 July 2021

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-72807-6

Online ISBN : 978-3-030-72808-3

eBook Packages : Social Sciences Social Sciences (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

How to present limitations in research

Last updated

30 January 2024

Reviewed by

Limitations don’t invalidate or diminish your results, but it’s best to acknowledge them. This will enable you to address any questions your study failed to answer because of them.

In this guide, learn how to recognize, present, and overcome limitations in research.

• What is a research limitation?

Research limitations are weaknesses in your research design or execution that may have impacted outcomes and conclusions. Uncovering limitations doesn’t necessarily indicate poor research design—it just means you encountered challenges you couldn’t have anticipated that limited your research efforts.

Does basic research have limitations?

Basic research aims to provide more information about your research topic. It requires the same standard research methodology and data collection efforts as any other research type, and it can also have limitations.

• Common research limitations

Researchers encounter common limitations when embarking on a study. Limitations can occur in relation to the methods you apply or the research process you design. They could also be connected to you as the researcher.

Methodology limitations

Not having access to data or reliable information can impact the methods used to facilitate your research. A lack of data or reliability may limit the parameters of your study area and the extent of your exploration.

Your sample size may also be affected because you won’t have any direction on how big or small it should be and who or what you should include. Having too few participants won’t adequately represent the population or groups of people needed to draw meaningful conclusions.

Research process limitations

The study’s design can impose constraints on the process. For example, as you’re conducting the research, issues may arise that don’t conform to the data collection methodology you developed. You may not realize until well into the process that you should have incorporated more specific questions or comprehensive experiments to generate the data you need to have confidence in your results.

Constraints on resources can also have an impact. Being limited on participants or participation incentives may limit your sample sizes. Insufficient tools, equipment, and materials to conduct a thorough study may also be a factor.

Common researcher limitations

Here are some of the common researcher limitations you may encounter:

Time: some research areas require multi-year longitudinal approaches, but you might not be able to dedicate that much time. Imagine you want to measure how much memory a person loses as they age. This may involve conducting multiple tests on a sample of participants over 20–30 years, which may be impossible.

Bias: researchers can consciously or unconsciously apply bias to their research. Biases can contribute to relying on research sources and methodologies that will only support your beliefs about the research you’re embarking on. You might also omit relevant issues or participants from the scope of your study because of your biases.

Limited access to data : you may need to pay to access specific databases or journals that would be helpful to your research process. You might also need to gain information from certain people or organizations but have limited access to them. These cases require readjusting your process and explaining why your findings are still reliable.

• Why is it important to identify limitations?

Identifying limitations adds credibility to research and provides a deeper understanding of how you arrived at your conclusions.

Constraints may have prevented you from collecting specific data or information you hoped would prove or disprove your hypothesis or provide a more comprehensive understanding of your research topic.

However, identifying the limitations contributing to your conclusions can inspire further research efforts that help gather more substantial information and data.

• Where to put limitations in a research paper

A research paper is broken up into different sections that appear in the following order:

Introduction

Methodology

The discussion portion of your paper explores your findings and puts them in the context of the overall research. Either place research limitations at the beginning of the discussion section before the analysis of your findings or at the end of the section to indicate that further research needs to be pursued.

What not to include in the limitations section

Evidence that doesn’t support your hypothesis is not a limitation, so you shouldn’t include it in the limitation section. Don’t just list limitations and their degree of severity without further explanation.

• How to present limitations

You’ll want to present the limitations of your study in a way that doesn’t diminish the validity of your research and leave the reader wondering if your results and conclusions have been compromised.

Include only the limitations that directly relate to and impact how you addressed your research questions. Following a specific format enables the reader to develop an understanding of the weaknesses within the context of your findings without doubting the quality and integrity of your research.

Identify the limitations specific to your study

You don’t have to identify every possible limitation that might have occurred during your research process. Only identify those that may have influenced the quality of your findings and your ability to answer your research question.

Explain study limitations in detail

This explanation should be the most significant portion of your limitation section.

Link each limitation with an interpretation and appraisal of their impact on the study. You’ll have to evaluate and explain whether the error, method, or validity issues influenced the study’s outcome and how.

Propose a direction for future studies and present alternatives

In this section, suggest how researchers can avoid the pitfalls you experienced during your research process.

If an issue with methodology was a limitation, propose alternate methods that may help with a smoother and more conclusive research project. Discuss the pros and cons of your alternate recommendation.

Describe steps taken to minimize each limitation

You probably took steps to try to address or mitigate limitations when you noticed them throughout the course of your research project. Describe these steps in the limitation section.

• Limitation example

“Approaches like stem cell transplantation and vaccination in AD [Alzheimer’s disease] work on a cellular or molecular level in the laboratory. However, translation into clinical settings will remain a challenge for the next decade.”

The authors are saying that even though these methods showed promise in helping people with memory loss when conducted in the lab (in other words, using animal studies), more studies are needed. These may be controlled clinical trials, for example. 

However, the short life span of stem cells outside the lab and the vaccination’s severe inflammatory side effects are limitations. Researchers won’t be able to conduct clinical trials until these issues are overcome.

• How to overcome limitations in research

You’ve already started on the road to overcoming limitations in research by acknowledging that they exist. However, you need to ensure readers don’t mistake weaknesses for errors within your research design.

To do this, you’ll need to justify and explain your rationale for the methods, research design, and analysis tools you chose and how you noticed they may have presented limitations.

Your readers need to know that even when limitations presented themselves, you followed best practices and the ethical standards of your field. You didn’t violate any rules and regulations during your research process.

You’ll also want to reinforce the validity of your conclusions and results with multiple sources, methods, and perspectives. This prevents readers from assuming your findings were derived from a single or biased source.

• Learning and improving starts with limitations in research

Dealing with limitations with transparency and integrity helps identify areas for future improvements and developments. It’s a learning process, providing valuable insights into how you can improve methodologies, expand sample sizes, or explore alternate approaches to further support the validity of your findings.

Editor’s picks

Last updated: 11 January 2024

Last updated: 15 January 2024

Last updated: 25 November 2023

Last updated: 12 May 2023

Last updated: 30 April 2024

Last updated: 18 May 2023

Last updated: 10 April 2023

Latest articles

Related topics, .css-je19u9{-webkit-align-items:flex-end;-webkit-box-align:flex-end;-ms-flex-align:flex-end;align-items:flex-end;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-box-flex-wrap:wrap;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;row-gap:0;text-align:center;max-width:671px;}@media (max-width: 1079px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}}@media (max-width: 799px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}} decide what to .css-1kiodld{max-height:56px;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}@media (max-width: 1079px){.css-1kiodld{display:none;}} build next, decide what to build next.

Users report unexpectedly high data usage, especially during streaming sessions.

Users find it hard to navigate from the home page to relevant playlists in the app.

It would be great to have a sleep timer feature, especially for bedtime listening.

I need better filters to find the songs or artists I’m looking for.

Log in or sign up

Get started for free

• Translators
• Graphic Designers

Please enter the email address you used for your account. Your sign in information will be sent to your email address after it has been verified.

Exploring Scope and Delimitation in Academic Research

Academic research is a meticulous process that requires precise planning and clear boundaries. Two pivotal components in this process are the scope and delimitations of the study. The definitions and establishment of these parameters are instrumental in ensuring that the research is effective, manageable, and yields relevant results.

The "scope" of a research project refers to the areas that the study will cover. It is the breadth and depth of the investigation. It defines the subject matter, the geographical location, the time frame, and the issues that the study will explore. Essentially, the scope delineates what the researcher aims to cover in the study.

On the other hand, "delimitations" are the boundaries or limitations set by the researcher. They define what the study will not include. Delimitations could involve the choice of research methodology , the selection of respondents, the duration of the study, and more. They help in confining the study to a manageable size while excluding peripheral elements.

Understanding and correctly implementing scope and delimitations are vital to ensuring your research is well-defined and focused, facilitating higher accuracy and relevancy in your findings.

Importance of scope in research

"Scope" in research refers to the comprehensive extent of study—it outlines the parameters of what will be explored and addressed. It defines the topic of the research , the geographical region under study, the timeframe considered, and the issues that the study will address. The scope of a research project is vital because it determines the depth and breadth of your investigation.

Defining the scope of research is a fundamental step in the research process for several reasons. First, it provides a roadmap for the study, giving the researcher clear guidelines about what to include and exclude. Without a well-defined scope, research can become unmanageably vast or lose its focus.

Second, the scope ensures the research's relevance and applicability. It helps the researcher maintain a tight focus on the study's central question , ensuring that all aspects of the research contribute to answering this question. This focus aids in avoiding irrelevant diversions that could dilute the final conclusions.

Finally, a well-defined scope can help ensure the efficient use of resources. Research involves considerable time, effort, and often financial resources. By providing clear boundaries, the scope ensures these resources are utilized effectively without wasted effort on peripheral issues.

Suppose a research study is looking at the impacts of social media usage on mental health. If the scope is too broad—like examining all social media platforms' effects on all demographic groups worldwide—then the research can quickly become unwieldy and hard to manage. It would involve vast amounts of data, requiring considerable time, resources, and computational power to analyze effectively.

However, if the scope is narrowed down—such as investigating the impact of Instagram usage on the mental health of teenagers in a specific city over the past five years—the research becomes far more manageable. This specific focus allows for a more in-depth analysis and likely will provide more meaningful, actionable results. This example illustrates the importance of appropriately defining the scope of research for its successful execution.

Determining the scope of your research

Setting the scope of your research project is a critical and delicate task. Below are steps, tips, and common mistakes to avoid when determining the scope of your research:

Steps to define the scope

• Identify Your Topic: The first step involves identifying and understanding your research topic. This knowledge will serve as a basis for determining the breadth and depth of your study.
• Define Your Research Questions: The research questions are the heart of your study. They will help you determine the specific areas your research should cover.
• Establish Boundaries: Clearly establish the geographical, temporal, and topical boundaries of your research. These boundaries will guide the range of your study.
• Choose Your Methodology: Decide on the research methods you will use as these will directly impact the scope of your study.

Tips for a manageable scope

• Stay Focused: Stay concentrated on your research questions. Do not stray into areas that aren't directly relevant.
• Be Realistic: Consider the resources (time, money, manpower) available. Ensure your scope is feasible given these resources.
• Seek Guidance: Consult with your academic advisor or peers for feedback on your proposed scope.

Common mistakes to avoid

• Overly Broad Scope: Avoid setting an overly broad scope which could result in an unmanageable and unfocused study.
• Too Narrow Scope: Conversely, a scope that is too narrow may miss important aspects of the research topic.
• Ignoring Resources: Not taking into account available resources when setting the scope can lead to a project that is impossible to complete.

Defining the scope of your research is a delicate balance, requiring careful consideration of your research questions, resources, and the depth and breadth of investigation needed to answer these questions effectively.

Importance of delimitations in research

In the context of academic research, "delimitations" refers to the choices made by the researcher which define the boundaries of the study. These are the variables that lead the researcher to narrow the scope of the study from its potential vastness to a manageable size.

Delimitations might include the geographic area where the study is confined, the participants involved in the study, the methodology used, the time period considered, or the specific incidents or aspects the study will focus on. Essentially, delimitations are the self-imposed limitations on the scope of the study.

Defining the delimitations of a research project is crucial for several reasons. Firstly, they establish the context or setting in which the study occurs. This, in turn, allows for the work to be reproduced in a similar context for verification or refutation in future studies.

Secondly, delimitations provide a way to narrow the scope of the research to a manageable size, thus avoiding the pitfall of an overly ambitious project. They help researchers to stay focused on the main research questions and prevent diversion into irrelevant aspects.

Finally, clearly defined delimitations enhance the credibility of the research. They offer transparency about the research design and methodology, which adds to the validity of the results.

For instance, in a research study examining the impact of technology on student achievement in a certain district, examples of delimitations might include focusing only on public schools, considering only high school students, and confining the study to a particular school year. These choices help to focus the research and ensure its manageability. Therefore, delimitations play a pivotal role in structuring and guiding an effective and efficient research study.

Setting delimitations for your research

Establishing appropriate delimitations for your research project is an important part of research design. Here are some steps, guidelines, and common mistakes to consider when setting your research delimitations:

Steps to establish delimitations

• Identify the boundaries: Begin by deciding the geographical region, time period, and subject matter your research will cover.
• Determine Your Research Population: Identify the specific population your study will focus on. This could be based on age, profession, geographical location, etc.
• Choose Your Research Methods: Decide the specific methods you will use to collect and analyze data, as these decisions will also set limitations on your study.

Guidelines for choosing delimitations

• Align with Your Research Objectives: The delimitations should be in line with your research questions and objectives. They should help focus your study without detracting from its goals.
• Be Practical: Consider the resources available, including time, funds, and access to data. Your delimitations should be feasible given these constraints.
• Seek Input: Consult with your research advisor or peers. Their feedback can help ensure your delimitations are appropriate and well thought out.

Common errors to avoid:

• Unrealistic Delimitations: Be wary of setting delimitations that are too stringent or ambitious to be feasible given your resources and timeframe.
• Undefined Delimitations: Avoid leaving your delimitations vague or undefined. This can lead to scope creep, where your project expands beyond its initial plan, making it unmanageable.
• Ignoring Delimitations: Once set, stick to your delimitations. Deviating from them can lead to a loss of focus and can compromise the integrity of your results.

Setting delimitations is a crucial step in research planning. Properly defined delimitations can make your research project more manageable, maintain your focus, and ensure the effective use of your resources.

The interplay between scope and delimitations

The relationship between scope and delimitations in academic research is a dynamic and interdependent one. Each aspect serves to shape and refine the other, ultimately leading to a focused, feasible, and effective research design.

The scope of a research project describes the breadth and depth of the investigation—what it aims to cover and how far it intends to delve into the subject matter. The delimitations, on the other hand, identify the boundaries and constraints of the study—what it will not cover.

As such, the scope and delimitations of a research study are intimately connected. When the scope of a study is broad, the delimitations must be carefully considered to ensure the project remains manageable and focused. Conversely, when the scope is narrow, the delimitations might be less constraining, but they still play a critical role in defining the specificity of the research.

Balancing the scope and delimitations is crucial for an efficient research design. Too broad a scope without carefully defined delimitations can lead to a study that is unwieldy and lacks depth. On the other hand, a very narrow scope with overly rigid delimitations might result in a study that overlooks important aspects of the research topic.

Thus, researchers must strive to maintain a balance—establishing a scope that is wide enough to fully explore the research topic, but also setting appropriate delimitations to ensure the study remains feasible and focused. In doing so, the research will be well-structured and yield meaningful, relevant findings.

Role of scope and delimitations in research validity

Scope and delimitations are fundamental aspects of research design that directly influence the validity, reliability, and replicability of a study.

Research validity refers to the degree to which a study accurately reflects or measures the concept that the researcher intends to investigate. A well-defined scope is critical to research validity because it clearly delineates what the study will cover. This clear definition ensures that the research focuses on relevant aspects of the topic and that the findings accurately reflect the concept under investigation.

Similarly, carefully thought-out delimitations contribute to research validity by identifying what the study will not cover. This clarity helps to prevent the study from straying into irrelevant areas, ensuring that the research stays focused and relevant.

In addition to contributing to research validity, scope and delimitations also influence the reliability and replicability of a study. Reliability refers to the consistency of a study's results, while replicability refers to the ability of other researchers to repeat the study and obtain similar results.

A clearly defined scope makes a study more reliable by providing a detailed outline of the areas covered by the research. This clarity makes it more likely that the study will produce consistent results. Moreover, clearly defined delimitations enhance the replicability of a study by providing explicit boundaries for the research, which makes it easier for other researchers to repeat the study in a similar context.

In summary, a well-defined scope and carefully thought-out delimitations contribute significantly to the validity, reliability, and replicability of academic research. They ensure that the research is focused, that the findings are relevant and accurate, and that the study can be reliably repeated by other researchers.

Examples of scope and delimitation in well-known research

• The Milgram Experiment: Stanley Milgram's famous psychology experiment sought to understand obedience to authority figures. The scope of this study was clearly defined—it focused on how far individuals would go in obeying an instruction if it involved harming another person. However, delimitations were set to ensure manageability. Participants were delimited to male individuals, and the experiment was confined to a controlled laboratory setting. These delimitations allowed Milgram to manage the research effectively while maintaining the depth of his study on human behavior.
• The Framingham Heart Study: This ongoing cardiovascular study began in 1948 and is aimed at identifying common factors that contribute to cardiovascular disease. The scope of the research is broad, covering many aspects of lifestyle, medical history, and physical characteristics. However, the study set clear delimitations: it initially only involved adult residents of Framingham, Massachusetts. This geographical delimitation made this broad-scope study manageable and eventually yielded influential results that shaped our understanding of heart disease.
• The Marshmallow Test: This well-known study by Walter Mischel explored delayed gratification in children. The scope was clearly defined: the study aimed to understand the ability of children to delay gratification and how it related to future success. The delimitations of the study included the age of the participants (preschool children), the setting (a controlled experiment with a treat), and the measure of future success (academic achievement, ability to cope with stress, etc.). These delimitations helped keep the study focused and manageable.

In all these examples, the researchers set a clear scope to outline the focus of their studies and used delimitations to restrict the boundaries. This balance between scope and delimitation was key in conducting successful and influential research.

In academic research, defining the scope and delimitations is a pivotal step in designing a robust and effective study. The scope outlines the breadth and depth of the investigation, offering a clear direction for the research. Meanwhile, delimitations set the boundaries of the study, ensuring that the research remains focused and manageable. Together, they play a crucial role in enhancing the validity, reliability, and replicability of a study.

Understanding the interplay between scope and delimitations is key to conducting efficient research. A well-defined scope paired with thoughtfully set delimitations contribute to a study's feasibility and its potential to yield meaningful and applicable results. Mistakes in setting the scope and delimitations can lead to unwieldy, unfocused research or a study that overlooks important aspects of a research question.

Reviewing famous studies, like the Milgram Experiment, the Framingham Heart Study, and the Marshmallow Test, we observe how a balanced approach to setting scope and delimitations can result in influential and valuable findings. Therefore, researchers should give careful thought to defining the scope and delimitations of their studies, keeping in mind their research questions, available resources, and the need for balance between breadth and focus. By doing so, they pave the way for successful and impactful research outcomes.

Header image by Kübra Arslaner .

Related Posts

The Top 10 Academic Writing Myths

Elements of Research Design

• Academic Writing Advice
• All Blog Posts
• Writing Advice
• Admissions Writing Advice
• Book Writing Advice
• Short Story Advice
• Employment Writing Advice
• Business Writing Advice
• Web Content Advice
• Article Writing Advice
• Magazine Writing Advice
• Grammar Advice
• Dialect Advice
• Editing Advice
• Freelance Advice
• Legal Writing Advice
• Poetry Advice
• Graphic Design Advice
• Logo Design Advice
• Translation Advice
• Blog Reviews
• Short Story Award Winners
• Scholarship Winners

Need an academic editor before submitting your work?

• Around the Practice
• Between the Lines
• Contemporary Concepts
• Readout 360
• Insights from Experts at Mayo Clinic on Translating Evidence to Clinical Practice
• Optimizing Outcomes in Patients with HER2+ Metastatic Breast Cancer

• Conferences
• Publications
• Career Center

Geographical Information Systems: Applications and Limitations in Oncology Research

• Pamela R. Soulos, MPH
• James B. Yu, MD, MHS

This review describes the development and technical capabilities of GIS, potential applications of Geographical Informational Systems in cancer research, and the limitations of such work.

The relationship between geography and cancer incidence and treatment is a critical area of health outcomes research. Geographical information systems (GIS) are software packages designed to store and analyze data related to geographic locations. Although more commonly associated with the social sciences and urban planning, the use of GIS software in medical research has been increasing.[1] Moreover, since the 1999 establishment of the Geographical Informational Systems Special Interest Group (GISSIG) at the National Cancer Institute, oncology has been at the forefront of GIS-related health research. In this review, we discuss the potential applications and limitations of GIS software in oncology research. Our aims are to help clinicians and policy makers interpret studies generated using GIS, and to help clinical investigators implement GIS in future research.

Introduction

The relationship between geography and cancer incidence and treatment is a critical area of health outcomes research, and Geographical Informational Systems (GIS) is a tool increasingly used for research in this area. GIS software programs can describe the geographic distribution of oncology care. GIS can effectively evaluate the supply of treatment resources within a given area relative to cancer prevalence and, more importantly, monitor for potential geographic variations in cancer outcomes and highlight potential disparities in cancer care. Because of this, GIS is becoming increasingly relevant in policy-oriented research focused on optimizing limited oncology resources within large underserved areas. This review describes the development and technical capabilities of GIS, potential applications of GIS in cancer research, and the limitations of such work.

Development and Capabilities of GIS Software

Developed in the 1960s by Dr. Roger Tomlinson of the Canadian Department of Forestry and Rural Development, the first GIS was originally constructed for surveying and development in rural parts of Canada. The original program, known as "Canada Geographical Information System" (CGIS), eventually grew to encompass datasets that spanned the entire country and became a useful tool in resource planning and management.[2] By the 1970s, universities and government organizations around the world had developed alternative GIS programs, and GIS-based research emerged as an independent multidisciplinary field. In 1982, as personal computer use began to increase, the Environmental Systems Research Institute (ESRI) developed the first commercially available GIS package, known as ARC/INFO.[3] The advent of commercially available GIS packages drastically increased the use of GIS worldwide. Users began to create open-use, publicly editable map data. The influx of map data into the public domain has only increased in recent years with the advent of new GIS technology and has allowed GIS to permeate many research fields.[4]

The functional capabilities of GIS software are a combination of modern cartography and database management. GIS programs are traditionally comprised of at least three functional components. First, GIS software permits users to input data that corresponds to a geographic location. Second, GIS software enables users to create maps to visually display integrated georegistered data. Third, GIS has database capability that allows users to store and manipulate entered data and maps.

Although found commercially in a variety of different software packages, our discussion of the technical aspects of GIS software will be limited to ESRI's ArcGIS. ArcGIS is the GIS software most widely used in health services research; it is used by more than 300,000 organizations worldwide, including most federal agencies, all 50 United States health departments, and over 24,000 state and local governments.[5] Data is stored in ArcGIS using shapefile packages. Shapefile packages are storage formats that house geographic location and associated attribute data. For example, a standard shapefile package could contain cancer incidence data, organized by county within the United States.

Regardless of GIS software type, the functional capabilities of GIS software can be effectively used in many areas of health services research. In addition to the ability to store and display regional data, several other functionalities of GIS software are worth noting. GIS software allows users to create their own maps and geographic units that can be tailored to more accurately describe healthcare patterns. Examples of this method in practice are maps developed by the Dartmouth Institute of Health Policy and Clinical Practice describing Hospital Referral Regions and Hospital Service Areas.[6] GIS software also allows users to freely aggregate data between different geographic units of analysis. For example, users can combine the individual cancer incidences within counties to estimate the cancer incidence of an entire state.

Finally, GIS software allows for the quantitative analysis of geographic patterns through "spatial analysis." Prior to the advent of GIS software, mapping in medical research was mainly a qualitative examination of data. In "spatial analysis," GIS software allows users to find statistically significant geographic relationships. GIS can employ spatial autocorrelation to find statistically significant geographic clustering of a variable. For example, a user could employ spatial autocorrelation to test whether there is statistically significant clustering of cancer incidences among neighboring counties. [7] Additionally, GIS can calculate geographically weighted regressions (GWR) to evaluate spatial heterogeneity among independent and dependent variables. Finally, GIS can employ spatial interpolation to estimate the geographic distribution of a variable within a region given the geographic distribution of the variable in surrounding regions.

Potential Applications of GIS in Cancer Research

GIS software allows for a rigorous assessment of the unique role geography plays in clinical oncology. Many types of studies are possible using GIS, and our description and examples of the potential applications of GIS to cancer research are not meant to be exhaustive. However, four broad categories of study come to mind. The first type of study is also perhaps the simplest: a graphical representation of cancer incidence and mortality. The second type of study is one that centers on the distribution of oncology resources. The third type of study is an investigation of geographic practice patterns. The fourth type of study is one that takes these geographic variations in oncology resources and practice patterns and relates them to socioeconomic disparity in cancer outcomes.

GIS for the surveillance of cancer incidence and mortality

60 Racial Disparities in Hospitalization Outcomes Among Women With Metastatic Breast Cancer in the United States by Palliative Care Utilization

Stem Cell Transplant Approaches for Patients With Blood Cancers

Dr. Scadden discusses which blood cancer patients are eligible for stem cell transplantation, as well as the latest advances in the field.

62 The Impact of Genomic Assays on Breast Cancer Systemic Therapy Treatment Decisions in a Mostly Black Patient Population

Current Treatment of Esophageal Cancer and Promising Clinical Trials Underway

Dr. Ku discusses considerations in the treatment of esophageal cancer, current treatments, and the emerging role of immunotherapy.

64 Empowering Medical Students to Deliver Breast Health Education: A Community-Based Initiative

65 Racial Disparities in Treatment Patterns and Outcomes Among HER2-Low Metastatic Breast Cancer Patients Treated in US Community Oncology Practices

2 Commerce Drive Cranbury, NJ 08512

609-716-7777