generalizability of findings in research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Research bias
• What Is Generalizability? | Definition & Examples

What Is Generalizability? | Definition & Examples

Published on October 8, 2022 by Kassiani Nikolopoulou . Revised on March 3, 2023.

Generalizability is the degree to which you can apply the results of your study to a broader context. Research results are considered generalizable when the findings can be applied to most contexts, most people, most of the time.

Generalizability is determined by how representative your sample is of the target population . This is known as external validity .

Table of contents

What is generalizability, why is generalizability important, examples of generalizability, types of generalizability, how do you ensure generalizability in research, other types of research bias, frequently asked questions about generalizability.

The goal of research is to produce knowledge that can be applied as widely as possible. However, since it usually isn’t possible to analyze every member of a population, researchers make do by analyzing a portion of it, making statements about that portion.

To be able to apply these statements to larger groups, researchers must ensure that the sample accurately resembles the broader population.

In other words, the sample and the population must share the characteristics relevant to the research being conducted. When this happens, the sample is considered representative, and by extension, the study’s results are considered generalizable.

In general, a study has good generalizability when the results apply to many different types of people or different situations. In contrast, if the results can only be applied to a subgroup of the population or in a very specific situation, the study has poor generalizability.

Obtaining a representative sample is crucial for probability sampling . In contrast, studies using non-probability sampling designs are more concerned with investigating a few cases in depth, rather than generalizing their findings. As such, generalizability is the main difference between probability and non-probability samples.

There are three factors that determine the generalizability of your study in a probability sampling design:

• The randomness of the sample, with each research unit (e.g., person, business, or organization in your population) having an equal chance of being selected.
• How representative the sample is of your population.
• The size of your sample, with larger samples more likely to yield statistically significant results.

Generalizability is one of the three criteria (along with validity and reliability ) that researchers use to assess the quality of both quantitative and qualitative research. However, depending on the type of research, generalizability is interpreted and evaluated differently.

• In quantitative research , generalizability helps to make inferences about the population.
• In qualitative research , generalizability helps to compare the results to other results from similar situations.

Generalizability is crucial for establishing the validity and reliability of your study. In most cases, a lack of generalizability significantly narrows down the scope of your research—i.e., to whom the results can be applied.

Luckily, you have access to an anonymized list of all residents. This allows you to establish a sampling frame and proceed with simple random sampling . With the help of an online random number generator, you draw a simple random sample.

After obtaining your results (and prior to drawing any conclusions) you need to consider the generalizability of your results. Using an online sample calculator, you see that the ideal sample size is 341. With a sample of 341, you could be confident that your results are generalizable, but a sample of 100 is too small to be generalizable.

However, research results that cannot be generalized can still have value. It all depends on your research objectives .

You go to the museum for three consecutive Sundays to make observations.

Your observations yield valuable insights for the Getty Museum, and perhaps even for other museums with similar educational offerings.

There are two broad types of generalizability:

• Statistical generalizability, which applies to quantitative research
• Theoretical generalizability (also referred to as transferability ), which applies to qualitative research

Statistical generalizability is critical for quantitative research . The goal of quantitative research is to develop general knowledge that applies to all the units of a population while studying only a subset of these units (sample). Statistical generalization is achieved when you study a sample that accurately mirrors characteristics of the population. The sample needs to be sufficiently large and unbiased.

In qualitative research , statistical generalizability is not relevant. This is because qualitative research is primarily concerned with obtaining insights on some aspect of human experience, rather than data with solid statistical basis. By studying individual cases, researchers will try to get results that they can extend to similar cases. This is known as theoretical generalizability or transferability.

In order to apply your findings on a larger scale, you should take the following steps to ensure your research has sufficient generalizability.

• Define your population in detail. By doing so, you will establish what it is that you intend to make generalizations about. For example, are you going to discuss students in general, or students on your campus?
• Use random sampling . If the sample is truly random (i.e., everyone in the population is equally likely to be chosen for the sample), then you can avoid sampling bias and ensure that the sample will be representative of the population.
• Consider the size of your sample. The sample size must be large enough to support the generalization being made. If the sample represents a smaller group within that population, then the conclusions have to be downsized in scope.
• If you’re conducting qualitative research , try to reach a saturation point of important themes and categories. This way, you will have sufficient information to account for all aspects of the phenomenon under study.

After completing your research, take a moment to reflect on the generalizability of your findings. What didn’t go as planned and could impact your generalizability? For example, selection biases such as nonresponse bias can affect your results. Explain how generalizable your results are, as well as possible limitations, in the discussion section of your research paper .

Cognitive bias

• Confirmation bias
• Baader–Meinhof phenomenon
• Availability heuristic
• Halo effect
• Framing effect

Selection bias

• Sampling bias
• Ascertainment bias
• Attrition bias
• Self-selection bias
• Survivorship bias
• Nonresponse bias
• Undercoverage bias
• Hawthorne effect
• Observer bias
• Omitted variable bias
• Publication bias
• Pygmalion effect
• Recall bias
• Social desirability bias
• Placebo effect

Generalizability is important because it allows researchers to make inferences for a large group of people, i.e., the target population, by only studying a part of it (the sample ).

I nternal validity is the degree of confidence that the causal relationship you are testing is not influenced by other factors or variables .

External validity is the extent to which your results can be generalized to other contexts.

The validity of your experiment depends on your experimental design .

In the discussion , you explore the meaning and relevance of your research results , explaining how they fit with existing research and theory. Discuss:

• Your  interpretations : what do the results tell us?
• The  implications : why do the results matter?
• The  limitation s : what can’t the results tell us?

Scope of research is determined at the beginning of your research process , prior to the data collection stage. Sometimes called “scope of study,” your scope delineates what will and will not be covered in your project. It helps you focus your work and your time, ensuring that you’ll be able to achieve your goals and outcomes.

Defining a scope can be very useful in any research project, from a research proposal to a thesis or dissertation . A scope is needed for all types of research: quantitative , qualitative , and mixed methods .

To define your scope of research, consider the following:

• Budget constraints or any specifics of grant funding
• Your proposed timeline and duration
• Specifics about your population of study, your proposed sample size , and the research methodology you’ll pursue
• Any inclusion and exclusion criteria
• Any anticipated control , extraneous , or confounding variables that could bias your research if not accounted for properly.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Nikolopoulou, K. (2023, March 03). What Is Generalizability? | Definition & Examples. Scribbr. Retrieved April 2, 2024, from https://www.scribbr.com/research-bias/generalizability/

Is this article helpful?

Kassiani Nikolopoulou

Other students also liked, external validity | definition, types, threats & examples, what is probability sampling | types & examples, what is quantitative research | definition, uses & methods.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Research bias
• What Is Generalisability? | Definition & Examples

What Is Generalisability? | Definition & Examples

Published on 10 October 2022 by Kassiani Nikolopoulou . Revised on 3 March 2023.

Generalisability is the degree to which you can apply the results of your study to a broader context. Research results are considered generalisable when the findings can be applied to most contexts, most people, most of the time.

Generalisability is determined by how representative your sample is of the target population . This is known as external validity .

Table of contents

What is generalisability, why is generalisability important, examples of generalisability, types of generalisability, how do you ensure generalisability in research, other types of research bias, frequently asked questions about generalisability.

The goal of research is to produce knowledge that can be applied as widely as possible. However, since it usually isn’t possible to analyse every member of a population, researchers make do by analysing a portion of it, making statements about that portion.

To be able to apply these statements to larger groups, researchers must ensure that the sample accurately resembles the broader population.

In other words, the sample and the population must share the characteristics relevant to the research being conducted. When this happens, the sample is considered representative, and by extension, the study’s results are considered generalisable.

In general, a study has good generalisability when the results apply to many different types of people or different situations. In contrast, if the results can only be applied to a subgroup of the population or in a very specific situation, the study has poor generalisability.

Obtaining a representative sample is crucial for probability sampling . In contrast, studies using non-probability sampling designs are more concerned with investigating a few cases in depth, rather than generalising their findings. As such, generalisability is the main difference between probability and non-probability samples.

There are three factors that determine the generalisability of your study in a probability sampling design:

• The randomness of the sample, with each research unit (e.g., person, business, or organisation in your population) having an equal chance of being selected.
• How representative the sample is of your population.
• The size of your sample, with larger samples more likely to yield statistically significant results.

Generalisability is one of the three criteria (along with validity and reliability ) that researchers use to assess the quality of both quantitative and qualitative research. However, depending on the type of research, generalisability is interpreted and evaluated differently.

• In quantitative research , generalisability helps to make inferences about the population.
• In qualitative research , generalisability helps to compare the results to other results from similar situations.

Generalisability is crucial for establishing the validity and reliability of your study. In most cases, a lack of generalisability significantly narrows down the scope of your research—i.e., to whom the results can be applied.

Luckily, you have access to an anonymised list of all residents. This allows you to establish a sampling frame and proceed with simple random sampling . With the help of an online random number generator, you draw a simple random sample.

After obtaining your results (and prior to drawing any conclusions) you need to consider the generalisability of your results. Using an online sample calculator, you see that the ideal sample size is 341. With a sample of 341, you could be confident that your results are generalisable, but a sample of 100 is too small to be generalisable.

However, research results that cannot be generalised can still have value. It all depends on your research objectives .

You go to the museum for three consecutive Sundays to make observations.

Your observations yield valuable insights for the Getty Museum, and perhaps even for other museums with similar educational offerings.

There are two broad types of generalisability:

• Statistical generalisability, which applies to quantitative research
• Theoretical generalisability (also referred to as transferability ), which applies to qualitative research

Statistical generalisability is critical for quantitative research . The goal of quantitative research is to develop general knowledge that applies to all the units of a population while studying only a subset of these units (sample). Statistical generalisation is achieved when you study a sample that accurately mirrors characteristics of the population. The sample needs to be sufficiently large and unbiased.

In qualitative research , statistical generalisability is not relevant. This is because qualitative research is primarily concerned with obtaining insights on some aspect of human experience, rather than data with solid statistical basis. By studying individual cases, researchers will try to get results that they can extend to similar cases. This is known as theoretical generalisability or transferability.

In order to apply your findings on a larger scale, you should take the following steps to ensure your research has sufficient generalisability.

• Define your population in detail. By doing so, you will establish what it is that you intend to make generalisations about. For example, are you going to discuss students in general, or students on your campus?
• Use random sampling . If the sample is truly random (i.e., everyone in the population is equally likely to be chosen for the sample), then you can avoid sampling bias and ensure that the sample will be representative of the population.
• Consider the size of your sample. The sample size must be large enough to support the generalisation being made. If the sample represents a smaller group within that population, then the conclusions have to be downsized in scope.
• If you’re conducting qualitative research , try to reach a saturation point of important themes and categories. This way, you will have sufficient information to account for all aspects of the phenomenon under study.

After completing your research, take a moment to reflect on the generalisability of your findings. What didn’t go as planned and could impact your generalisability? For example, selection biases such as non-response bias can affect your results. Explain how generalisable your results are, as well as possible limitations, in the discussion section of your research paper .

Cognitive bias

• Confirmation bias
• Baader–Meinhof phenomenon
• Availability heuristic
• Halo effect
• Framing effect

Selection bias

• Sampling bias
• Ascertainment bias
• Attrition bias
• Self-selection bias
• Survivorship bias
• Nonresponse bias
• Undercoverage bias
• Hawthorne effect
• Observer bias
• Omitted variable bias
• Publication bias
• Pygmalion effect
• Recall bias
• Social desirability bias
• Placebo effect

I nternal validity is the degree of confidence that the causal relationship you are testing is not influenced by other factors or variables .

External validity is the extent to which your results can be generalised to other contexts.

The validity of your experiment depends on your experimental design .

Generalisability is important because it allows researchers to make inferences for a large group of people, i.e., the target population, by only studying a part of it (the sample ).

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Nikolopoulou, K. (2023, March 03). What Is Generalisability? | Definition & Examples. Scribbr. Retrieved 2 April 2024, from https://www.scribbr.co.uk/bias-in-research/generalisability/

Is this article helpful?

Kassiani Nikolopoulou

Other students also liked, external validity | types, threats & examples, what is probability sampling | types & examples, what is quantitative research | definition & methods.

Understanding Generalizability and Transferability

In this chapter, we discuss generalizabililty, transferability, and the interrelationship between the two. We also explain how these two aspects of research operate in different methodologies, demonstrating how researchers may apply these concepts throughout the research process.

Generalizability Overview

Generalizability is applied by researchers in an academic setting. It can be defined as the extension of research findings and conclusions from a study conducted on a sample population to the population at large. While the dependability of this extension is not absolute, it is statistically probable. Because sound generalizability requires data on large populations, quantitative research -- experimental for instance -- provides the best foundation for producing broad generalizability. The larger the sample population, the more one can generalize the results. For example, a comprehensive study of the role computers play in the writing process might reveal that it is statistically probable that students who do most of their composing on a computer will move chunks of text around more than students who do not compose on a computer.

Transferability Overview

Transferability is applied by the readers of research. Although generalizability usually applies only to certain types of quantitative methods, transferability can apply in varying degrees to most types of research . Unlike generalizability, transferability does not involve broad claims, but invites readers of research to make connections between elements of a study and their own experience. For instance, teachers at the high school level might selectively apply to their own classrooms results from a study demonstrating that heuristic writing exercises help students at the college level.

Interrelationships

Generalizability and transferability are important elements of any research methodology, but they are not mutually exclusive: generalizability, to varying degrees, rests on the transferability of research findings. It is important for researchers to understand the implications of these twin aspects of research before designing a study. Researchers who intend to make a generalizable claim must carefully examine the variables involved in the study. Among these are the sample of the population used and the mechanisms behind formulating a causal model. Furthermore, if researchers desire to make the results of their study transferable to another context, they must keep a detailed account of the environment surrounding their research, and include a rich description of that environment in their final report. Armed with the knowledge that the sample population was large and varied, as well as with detailed information about the study itself, readers of research can more confidently generalize and transfer the findings to other situations.

Generalizability

Generalizability is not only common to research, but to everyday life as well. In this section, we establish a practical working definition of generalizability as it is applied within and outside of academic research. We also define and consider three different types of generalizability and some of their probable applications. Finally, we discuss some of the possible shortcomings and limitations of generalizability that researchers must be aware of when constructing a study they hope will yield potentially generalizable results.

In many ways, generalizability amounts to nothing more than making predictions based on a recurring experience. If something occurs frequently, we expect that it will continue to do so in the future. Researchers use the same type of reasoning when generalizing about the findings of their studies. Once researchers have collected sufficient data to support a hypothesis, a premise regarding the behavior of that data can be formulated, making it generalizable to similar circumstances. Because of its foundation in probability, however, such a generalization cannot be regarded as conclusive or exhaustive.

While generalizability can occur in informal, nonacademic settings, it is usually applied only to certain research methods in academic studies. Quantitative methods allow some generalizability. Experimental research, for example, often produces generalizable results. However, such experimentation must be rigorous in order for generalizable results to be found.

An example of generalizability in everyday life involves driving. Operating an automobile in traffic requires that drivers make assumptions about the likely outcome of certain actions. When approaching an intersection where one driver is preparing to turn left, the driver going straight through the intersection assumes that the left-turning driver will yield the right of way before turning. The driver passing through the intersection applies this assumption cautiously, recognizing the possibility that the other driver might turn prematurely.

American drivers also generalize that everyone will drive on the right hand side of the road. Yet if we try to generalize this assumption to other settings, such as England, we will be making a potentially disastrous mistake. Thus, it is obvious that generalizing is necessary for forming coherent interpretations in many different situations, but we do not expect our generalizations to operate the same way in every circumstance. With enough evidence we can make predictions about human behavior, yet we must simultaneously recognize that our assumptions are based on statistical probability.

Consider this example of generalizable research in the field of English studies. A study on undergraduate instructor evaluations of composition instructors might reveal that there is a strong correlation between the grade students are expecting to earn in a course and whether they give their instructor high marks. The study might discover that 95% of students who expect to receive a "C" or lower in their class give their instructor a rating of "average" or below. Therefore, there would be a high probability that future students expecting a "C" or lower would not give their instructor high marks. However, the results would not necessarily be conclusive. Some students might defy the trend. In addition, a number of different variables could also influence students' evaluations of an instructor, including instructor experience, class size, and relative interest in a particular subject. These variables -- and others -- would have to be addressed in order for the study to yield potentially valid results. However, even if virtually all variables were isolated, results of the study would not be 100% conclusive. At best, researchers can make educated predictions of future events or behaviors, not guarantee the prediction in every case. Thus, before generalizing, findings must be tested through rigorous experimentation, which enables researchers to confirm or reject the premises governing their data set.

Considerations

There are three types of generalizability that interact to produce probabilistic models. All of them involve generalizing a treatment or measurement to a population outside of the original study. Researchers who wish to generalize their claims should try to apply all three forms to their research, or the strength of their claims will be weakened (Runkel & McGrath, 1972).

In one type of generalizability, researchers determine whether a specific treatment will produce the same results in different circumstances. To do this, they must decide if an aspect within the original environment, a factor beyond the treatment, generated the particular result. This will establish how flexibly the treatment adapts to new situations. Higher adaptability means that the treatment is generalizable to a greater variety of situations. For example, imagine that a new set of heuristic prewriting questions designed to encourage freshman college students to consider audience more fully works so well that the students write thoroughly developed rhetorical analyses of their target audiences. To responsibly generalize that this heuristic is effective, a researcher would need to test the same prewriting exercise in a variety of educational settings at the college level, using different teachers, students, and environments. If the same positive results are produced, the treatment is generalizable.

A second form of generalizability focuses on measurements rather than treatments. For a result to be considered generalizable outside of the test group, it must produce the same results with different forms of measurement. In terms of the heuristic example above, the findings will be more generalizable if the same results are obtained when assessed "with questions having a slightly different wording, or when we use a six-point scale instead of a nine-point scale" (Runkel & McGrath, 1972, p.46).

A third type of generalizability concerns the subjects of the test situation. Although the results of an experiment may be internally valid, that is, applicable to the group tested, in many situations the results cannot be generalized beyond that particular group. Researchers who hope to generalize their results to a larger population should ensure that their test group is relatively large and randomly chosen. However, researchers should consider the fact that test populations of over 10,000 subjects do not significantly increase generalizability (Firestone,1993).

Potential Limitations

No matter how carefully these three forms of generalizability are applied, there is no absolute guarantee that the results obtained in a study will occur in every situation outside the study. In order to determine causal relationships in a test environment, precision is of utmost importance. Yet if researchers wish to generalize their findings, scope and variance must be emphasized over precision. Therefore, it becomes difficult to test for precision and generalizability simultaneously, since a focus on one reduces the reliability of the other. One solution to this problem is to perform a greater number of observations, which has a dual effect: first, it increases the sample population, which heightens generalizability; second, precision can be reasonably maintained because the random errors between observations will average out (Runkel and McGrath, 1972).

Transferability

Transferability describes the process of applying the results of research in one situation to other similar situations. In this section, we establish a practical working definition of transferability as it's applied within and outside of academic research. We also outline important considerations researchers must be aware of in order to make their results potentially transferable, as well as the critical role the reader plays in this process. Finally, we discuss possible shortcomings and limitations of transferability that researchers must be aware of when planning and conducting a study that will yield potentially transferable results.

Transferability is a process performed by readers of research. Readers note the specifics of the research situation and compare them to the specifics of an environment or situation with which they are familiar. If there are enough similarities between the two situations, readers may be able to infer that the results of the research would be the same or similar in their own situation. In other words, they "transfer" the results of a study to another context. To do this effectively, readers need to know as much as possible about the original research situation in order to determine whether it is similar to their own. Therefore, researchers must supply a highly detailed description of their research situation and methods.

Results of any type of research method can be applied to other situations, but transferability is most relevant to qualitative research methods such as ethnography and case studies. Reports based on these research methods are detailed and specific. However, because they often consider only one subject or one group, researchers who conduct such studies seldom generalize the results to other populations. The detailed nature of the results, however, makes them ideal for transferability.

Transferability is easy to understand when you consider that we are constantly applying this concept to aspects of our daily lives. If, for example, you are an inexperienced composition instructor and you read a study in which a veteran writing instructor discovered that extensive prewriting exercises helped students in her classes come up with much more narrowly defined paper topics, you could ask yourself how much the instructor's classroom resembled your own. If there were many similarities, you might try to draw conclusions about how increasing the amount of prewriting your students do would impact their ability to arrive at sufficiently narrow paper topics. In doing so, you would be attempting to transfer the composition researcher's techniques to your own classroom.

An example of transferable research in the field of English studies is Berkenkotter, Huckin, and Ackerman's (1988) study of a graduate student in a rhetoric Ph.D. program. In this case study, the researchers describe in detail a graduate student's entrance into the language community of his academic program, and particularly his struggle learning the writing conventions of this community. They make conclusions as to why certain things might have affected the graduate student, "Nate," in certain ways, but they are unable to generalize their findings to all graduate students in rhetoric Ph.D. programs. It is simply one study of one person in one program. However, from the level of detail the researchers provide, readers can take certain aspects of Nate's experience and apply them to other contexts and situations. This is transferability. First-year graduate students who read the Berkenhotter, Huckin, and Ackerman study may recognize similarities in their own situation while professors may recognize difficulties their students are having and understand these difficulties a bit better. The researchers do not claim that their results apply to other situations. Instead, they report their findings and make suggestions about possible causes for Nate's difficulties and eventual success. Readers then look at their own situation and decide if these causes may or may not be relevant.

When designing a study researchers have to consider their goals: Do they want to provide limited information about a broad group in order to indicate trends or patterns? Or do they want to provide detailed information about one person or small group that might suggest reasons for a particular behavior? The method they choose will determine the extent to which their results can be transferred since transferability is more applicable to certain kinds of research methods than others.

Thick Description: When writing up the results of a study, it is important that the researcher provide specific information about and a detailed description of her subject(s), location, methods, role in the study, etc. This is commonly referred to as "thick description" of methods and findings; it is important because it allows readers to make an informed judgment about whether they can transfer the findings to their own situation. For example, if an educator conducts an ethnography of her writing classroom, and finds that her students' writing improved dramatically after a series of student-teacher writing conferences, she must describe in detail the classroom setting, the students she observed, and her own participation. If the researcher does not provide enough detail, it will be difficult for readers to try the same strategy in their own classrooms. If the researcher fails to mention that she conducted this research in a small, upper-class private school, readers may transfer the results to a large, inner-city public school expecting a similar outcome.

The Reader's Role: The role of readers in transferability is to apply the methods or results of a study to their own situation. In doing so, readers must take into account differences between the situation outlined by the researcher and their own. If readers of the Berkenhotter, Huckin, and Ackerman study are aware that the research was conducted in a small, upper-class private school, but decide to test the method in a large inner-city public school, they must make adjustments for the different setting and be prepared for different results.

Likewise, readers may decide that the results of a study are not transferable to their own situation. For example, if a study found that watching more than 30 hours of television a week resulted in a worse GPA for graduate students in physics, graduate students in broadcast journalism may conclude that these results do not apply to them.

Readers may also transfer only certain aspects of the study and not the entire conclusion. For example, in the Berkenhotter, Huckin, and Ackerman study, the researchers suggest a variety of reasons for why the graduate student studied experienced difficulties adjusting to his Ph.D. program. Although composition instructors cannot compare "Nate" to first-year college students in their composition class, they could ask some of the same questions about their own class, offering them insight into some of the writing difficulties the first-year undergraduates are experiencing. It is up to readers to decide what findings are important and which may apply to their own situation; if researchers fulfill their responsibility to provide "thick description," this decision is much easier to make.

Understanding research results can help us understand why and how something happens. However, many researchers believe that such understanding is difficult to achieve in relation to human behaviors which they contend are too difficult to understand and often impossible to predict. "Because of the many and varied ways in which individuals differ from each other and because these differences change over time, comprehensive and definitive experiments in the social sciences are not possible...the most we can ever realistically hope to achieve in educational research is not prediction and control but rather only temporary understanding" (Cziko, 1993, p. 10).

Cziko's point is important because transferability allows for "temporary understanding." Instead of applying research results to every situation that may occur in the future, we can apply a similar method to another, similar situation, observe the new results, apply a modified version to another situation, and so on. Transferability takes into account the fact that there are no absolute answers to given situations; rather, every individual must determine their own best practices. Transferring the results of research performed by others can help us develop and modify these practices. However, it is important for readers of research to be aware that results cannot always be transferred; a result that occurs in one situation will not necessarily occur in a similar situation. Therefore, it is critical to take into account differences between situations and modify the research process accordingly.

Although transferability seems to be an obvious, natural, and important method for applying research results and conclusions, it is not perceived as a valid research approach in some academic circles. Perhaps partly in response to critics, in many modern research articles, researchers refer to their results as generalizable or externally valid. Therefore, it seems as though they are not talking about transferability. However, in many cases those same researchers provide direction about what points readers may want to consider, but hesitate to make any broad conclusions or statements. These are characteristics of transferable results.

Generalizability is actually, as we have seen, quite different from transferability. Unfortunately, confusion surrounding these two terms can lead to misinterpretation of research results. Emphasis on the value of transferable results -- as well as a clear understanding among researchers in the field of English of critical differences between the conditions under which research can be generalized, transferred, or, in some cases, both generalized and transferred -- could help qualitative researchers avoid some of the criticisms launched by skeptics who question the value of qualitative research methods.

Generalizability and Transferability: Synthesis

Generalizability allows us to form coherent interpretations in any situation, and to act purposefully and effectively in daily life. Transferability gives us the opportunity to sort through given methods and conclusions to decide what to apply to our own circumstances. In essence, then, both generalizability and transferability allow us to make comparisons between situations. For example, we can generalize that most people in the United States will drive on the right side of the road, but we cannot transfer this conclusion to England or Australia without finding ourselves in a treacherous situation. It is important, therefore, to always consider context when generalizing or transferring results.

Whether a study emphasizes transferability or generalizability is closely related to the goals of the researcher and the needs of the audience. Studies done for a magazine such as Time or a daily newspaper tend towards generalizability, since the publishers want to provide information relevant to a large portion of the population. A research project pointed toward a small group of specialists studying a similar problem may emphasize transferability, since specialists in the field have the ability to transfer aspects of the study results to their own situations without overt generalizations provided by the researcher. Ultimately, the researcher's subject, audience, and goals will determine the method the researcher uses to perform a study, which will then determine the transferability or generalizability of the results.

A Comparison of Generalizability and Transferability

Although generalizability has been a preferred method of research for quite some time, transferability is relatively a new idea. In theory, however, it has always accompanied research issues. It is important to note that generalizability and transferability are not necessarily mutually exclusive; they can overlap.

From an experimental study to a case study, readers transfer the methods, results, and ideas from the research to their own context. Therefore, a generalizable study can also be transferable. For example, a researcher may generalize the results of a survey of 350 people in a university to the university population as a whole; readers of the results may apply, or transfer, the results to their own situation. They will ask themselves, basically, if they fall into the majority or not. However, a transferable study is not always generalizable. For example, in case studies , transferability allows readers the option of applying results to outside contexts, whereas generalizability is basically impossible because one person or a small group of people is not necessarily representative of the larger population.

Controversy, Worth, and Function

Research in the natural sciences has a long tradition of valuing empirical studies; experimental investigation has been considered "the" way to perform research. As social scientists adapted the methods of natural science research to their own needs, they adopted this preference for empirical research. Therefore, studies that are generalizable have long been thought to be more worthwhile; the value of research was often determined by whether a study was generalizable to a population as a whole. However, more and more social scientists are realizing the value of using a variety of methods of inquiry, and the value of transferability is being recognized.

It is important to recognize that generalizability and transferability do not alone determine a study's worth. They perform different functions in research, depending on the topic and goals of the researcher. Where generalizable studies often indicate phenomena that apply to broad categories such as gender or age, transferability can provide some of the how and why behind these results.

However, there are weaknesses that must be considered. Researchers can study a small group that is representative of a larger group and claim that it is likely that their results are applicable to the larger group, but it is impossible for them to test every single person in the larger group. Their conclusions, therefore, are only valid in relation to their own studies. Another problem is that a non-representative group can lead to a faulty generalization. For example, a study of composition students'; revision capabilities which compared students' progress made during a semester in a computer classroom with progress exhibited by students in a traditional classroom might show that computers do aid students in the overall composing process. However, if it were discovered later that an unusually high number of students in the traditional classrooms suffered from substance abuse problems outside of the classroom, the population studied would not be considered representative of the student population as a whole. Therefore, it would be problematic to generalize the results of the study to a larger student population.

In the case of transferability, readers need to know as much detail as possible about a research situation in order to accurately transfer the results to their own. However, it is impossible to provide an absolutely complete description of a situation, and missing details may lead a reader to transfer results to a situation that is not entirely similar to the original one.

Applications to Research Methods

The degree to which generalizability and transferability are applicable differs from methodology to methodology as well as from study to study. Researchers need to be aware of these degrees so that results are not undermined by over-generalizations, and readers need to ensure that they do not read researched results in such a way that the results are misapplied or misinterpreted.

Applications of Transferability and Generalizability: Case Study

Research Design Case studies examine individuals or small groups within a specific context. Research is typically gathered through qualitative means: interviews, observations, etc. Data is usually analyzed either holistically or by coding methods.

Assumptions In research involving case studies, a researcher typically assumes that the results will be transferable. Generalizing is difficult or impossible because one person or small group cannot represent all similar groups or situations. For example, one group of beginning writing students in a particular classroom cannot represent all beginning student writers. Also, conclusions drawn in case studies are only about the participants being observed. With rare exceptions, case studies are not meant to establish cause/effect relationships between variables. The results of a case study are transferable in that researchers "suggest further questions, hypotheses, and future implications," and present the results as "directions and questions" (Lauer & Asher 32).

Example In order to illustrate the writing skills of beginning college writers, a researcher completing a case study might single out one or more students in a composition classroom and set about talking to them about how they judge their own writing as well as reading actual papers, setting up criteria for judgment, and reviewing paper grades/teacher interpretation.

Results of a Study In presenting the results of the previous example, a researcher should define the criteria that were established in order to determine what the researcher meant by "writing skills," provide noteworthy quotes from student interviews, provide other information depending on the kinds of research methods used (e.g., surveys, classroom observation, collected writing samples), and include possibilities for furthering this type of research. Readers are then able to assess for themselves how the researcher's observations might be transferable to other writing classrooms.

Applications of Transferability and Generalizability: Ethnography

Research Design Ethnographies study groups and/or cultures over a period of time. The goal of this type of research is to comprehend the particular group/culture through observer immersion into the culture or group. Research is completed through various methods, which are similar to those of case studies, but since the researcher is immersed within the group for an extended period of time, more detailed information is usually collected during the research. (Jonathon Kozol's "There Are No Children Here" is a good example of this.)

Assumptions As with case studies, findings of ethnographies are also considered to be transferable. The main goals of an ethnography are to "identify, operationally define, and interrelate variables" within a particular context, which ultimately produce detailed accounts or "thick descriptions" (Lauer & Asher 39). Unlike a case study, the researcher here discovers many more details. Results of ethnographies should "suggest variables for further investigation" and not generalize beyond the participants of a study (Lauer & Asher 43). Also, since analysts completing this type of research tend to rely on multiple methods to collect information (a practice also referred to as triangulation), their results typically help create a detailed description of human behavior within a particular environment.

Example The Iowa Writing Program has a widespread reputation for producing excellent writers. In order to begin to understand their training, an ethnographer might observe students throughout their degree program. During this time, the ethnographer could examine the curriculum, follow the writing processes of individual writers, and become acquainted with the writers and their work. By the end of a two year study, the researcher would have a much deeper understanding of the unique and effective features of the program.

Results of a Study Obviously, the Iowa Writing Program is unique, so generalizing any results to another writing program would be problematic. However, an ethnography would provide readers with insights into the program. Readers could ask questions such as: what qualities make it strong and what is unique about the writers who are trained within the program? At this point, readers could attempt to "transfer" applicable knowledge and observations to other writing environments.

Applications of Transferability and Generalizability: Experimental Research

Research Design A researcher working within this methodology creates an environment in which to observe and interpret the results of a research question. A key element in experimental research is that participants in a study are randomly assigned to groups. In an attempt to create a causal model (i.e., to discover the causal origin of a particular phenomenon), groups are treated differently and measurements are conducted to determine if different treatments appear to lead to different effects.

Assumptions Experimental research is usually thought to be generalizable. This methodology explores cause/effect relationships through comparisons among groups (Lauer & Asher 152). Since participants are randomly assigned to groups, and since most experiments involve enough individuals to reasonably approximate the populations from which individual participants are drawn, generalization is justified because "over a large number of allocations, all the groups of subjects will be expected to be identical on all variables" (155).

Example A simplified example: Six composition classrooms are randomly chosen (as are the students and instructors) in which three instructors incorporate the use of electronic mail as a class activity and three do not. When students in the first three classes begin discussing their papers through e-mail and, as a result, make better revisions to their papers than students in the other three classes, a researcher is likely to conclude that incorporating e-mail within a writing classroom improves the quality of students' writing.

Results of a Study Although experimental research is based on cause/effect relationships, "certainty" can never be obtained, but rather results are "probabilistic" (Lauer and Asher 161). Depending on how the researcher has presented the results, they are generalizable in that the students were selected randomly. Since the quality of writing improved with the use of e-mail within all three classrooms, it is probable that e-mail is the cause of the improvement. Readers of this study would transfer the results when they sorted out the details: Are these students representative of a group of students with which the reader is familiar? What types of previous writing experiences have these students had? What kind of writing was expected from these students? The researcher must have provided these details in order for the results to be transferable.

Applications of Transferability and Generalizability: Survey

Research Design The goal of a survey is to gain specific information about either a specific group or a representative sample of a particular group. Survey respondents are asked to respond to one or more of the following kinds of items: open-ended questions, true-false questions, agree-disagree (or Likert) questions, rankings, ratings, and so on. Results are typically used to understand the attitudes, beliefs, or knowledge of a particular group.

Assumptions Assuming that care has been taken in the development of the survey items and selection of the survey sample and that adequate response rates have been achieved, surveys results are generalizable. Note, however, that results from surveys should be generalized only to the population from which the survey results were drawn.

Example For instance, a survey of Colorado State University English graduate students undertaken to determine how well French philosopher/critic Jacques Derrida is understood before and after students take a course in critical literary theory might inform professors that, overall, Derrida's concepts are understood and that CSU's literary theory class, E615, has helped students grasp Derrida's ideas.

Results of a Study The generalizability of surveys depends on several factors. Whether distributed to a mass of people or a select few, surveys are of a "personal nature and subject to distortion." Survey respondents may or may not understand the questions being asked of them. Depending on whether or not the survey designer is nearby, respondents may or may not have the opportunity to clarify their misunderstandings.

It is also important to keep in mind that errors can occur at the development and processing levels. A researcher may inadequately pose questions (that is, not ask the right questions for the information being sought), disrupt the data collection (surveying certain people and not others), and distort the results during the processing (misreading responses and not being able to question the participant, etc.). One way to avoid these kinds of errors is for researchers to examine other studies of a similar nature and compare their results with results that have been obtained in previous studies. This way, any large discrepancies will be exposed. Depending on how large those discrepancies are and what the context of the survey is, the results may or may not be generalizable. For example, if an improved understanding of Derrida is apparent after students complete E615, it can be theorized that E615 effectively teaches students the concepts of Derrida. Issues of transferability might be visible in the actual survey questions themselves; that is, they could provide critical background information readers might need to know in order to transfer the results to another context.

The Qualitative versus Quantitative Debate

In Miles and Huberman's 1994 book Qualitative Data Analysis , quantitative researcher Fred Kerlinger is quoted as saying, "There's no such thing as qualitative data. Everything is either 1 or 0" (p. 40). To this another researcher, D. T. Campbell, asserts "all research ultimately has a qualitative grounding" (p. 40). This back and forth banter among qualitative and quantitative researchers is "essentially unproductive" according to Miles and Huberman. They and many other researchers agree that these two research methods need each other more often than not. However, because typically qualitative data involves words and quantitative data involves numbers, there are some researchers who feel that one is better (or more scientific) than the other. Another major difference between the two is that qualitative research is inductive and quantitative research is deductive. In qualitative research, a hypothesis is not needed to begin research. However, all quantitative research requires a hypothesis before research can begin.

Another major difference between qualitative and quantitative research is the underlying assumptions about the role of the researcher. In quantitative research, the researcher is ideally an objective observer that neither participates in nor influences what is being studied. In qualitative research, however, it is thought that the researcher can learn the most about a situation by participating and/or being immersed in it. These basic underlying assumptions of both methodologies guide and sequence the types of data collection methods employed.

Although there are clear differences between qualitative and quantitative approaches, some researchers maintain that the choice between using qualitative or quantitative approaches actually has less to do with methodologies than it does with positioning oneself within a particular discipline or research tradition. The difficulty of choosing a method is compounded by the fact that research is often affiliated with universities and other institutions. The findings of research projects often guide important decisions about specific practices and policies. The choice of which approach to use may reflect the interests of those conducting or benefitting from the research and the purposes for which the findings will be applied. Decisions about which kind of research method to use may also be based on the researcher's own experience and preference, the population being researched, the proposed audience for findings, time, money, and other resources available (Hathaway, 1995).

Some researchers believe that qualitative and quantitative methodologies cannot be combined because the assumptions underlying each tradition are so vastly different. Other researchers think they can be used in combination only by alternating between methods: qualitative research is appropriate to answer certain kinds of questions in certain conditions and quantitative is right for others. And some researchers think that both qualitative and quantitative methods can be used simultaneously to answer a research question.

To a certain extent, researchers on all sides of the debate are correct: each approach has its drawbacks. Quantitative research often "forces" responses or people into categories that might not "fit" in order to make meaning. Qualitative research, on the other hand, sometimes focuses too closely on individual results and fails to make connections to larger situations or possible causes of the results. Rather than discounting either approach for its drawbacks, though, researchers should find the most effective ways to incorporate elements of both to ensure that their studies are as accurate and thorough as possible.

It is important for researchers to realize that qualitative and quantitative methods can be used in conjunction with each other. In a study of computer-assisted writing classrooms, Snyder (1995) employed both qualitative and quantitative approaches. The study was constructed according to guidelines for quantitative studies: the computer classroom was the "treatment" group and the traditional pen and paper classroom was the "control" group. Both classes contained subjects with the same characteristics from the population sampled. Both classes followed the same lesson plan and were taught by the same teacher in the same semester. The only variable used was the computers. Although Snyder set this study up as an "experiment," she used many qualitative approaches to supplement her findings. She observed both classrooms on a regular basis as a participant-observer and conducted several interviews with the teacher both during and after the semester. However, there were several problems in using this approach: the strict adherence to the same syllabus and lesson plans for both classes and the restricted access of the control group to the computers may have put some students at a disadvantage. Snyder also notes that in retrospect she should have used case studies of the students to further develop her findings. Although her study had certain flaws, Snyder insists that researchers can simultaneously employ qualitative and quantitative methods if studies are planned carefully and carried out conscientiously.

Annotated Bibliography

Babbie, Earl R. (1979). The practice of social research . Belmont: Wadsworth Publishing Company, Inc.

A comprehensive review of social scientific research, including techniques for research. The logic behind social scientific research is discussed.

Berkenkotter, C., Huckin, T.N., & Ackerman, J. (1988). Conventions, conversations, and the writer: Case study of a student in a rhetoric Ph.D. program. Research in the Teaching of English 22 (1), 9-44.

Describes a case study of a beginning student in a Ph.D. program. Looks at the process of his entry into an academic discourse community.

Black, Susan. (1996). Redefining the teacher's role. Executive Educator,18 (8), 23-26.

Discusses the value of well-trained teacher-researchers performing research in their classrooms. Notes that teacher-research focuses on the particular; it does not look for broad, generalizable principles.

Blank, Steven C. (1984). Practical business research methods . Westport: AVI Publishing Company, Inc.

A comprehensive book of how to set up a research project, collect data, and reach and report conclusions.

Bridges, David. (1993). Transferable Skills: A Philosophical Perspective. Studies in Higher Education 18 (1), 43-51.

Transferability of skills in learning is discussed, focusing on the notions of cross-disciplinary, generic, core, and transferable skills and their role in the college curriculum.

Brookhart, Susan M. & Rusnak, Timothy G. (1993). A pedagogy of enrichment, not poverty: Successful lessons of exemplary urban teachers. Journal of Teacher Education, 44 (1), 17-27.

Reports the results of a study that explored the characteristics of effective urban teachers in Pittsburgh. Suggests that the results may be transferable to urban educators in other contexts.

Bryman, Alan. (1988). Quantity and quality in social research . Boston: Unwin Hyman Ltd.

Butcher, Jude. (1994, July). Cohort and case study components in teacher education research. Paper presented at the annual conference of the Australian Teacher Education Association, Brisbane, Queensland, Australia.

Argues that studies of teacher development will be more generalizable if a broad set of methods are used to collect data, if the data collected is both extensive and intensive, and if the methods used take into account the differences in people and situations being studied.

Carter, Duncan. (1993). Critical thinking for writers: Transferable skills or discipline-specific strategies? Composition Studies/Freshman English News, 21 (1), 86-93.

Questions the context-dependency of critical thinking, and whether critical thinking skills are transferable to writing tasks.

Carter, Kathy. (1993). The place of story in the study of teaching and teacher education. Educational Researcher, 22 (1), 5-12.

Discusses the advantages of story-telling in teaching and teacher education, but cautions instructors, who are currently unfamiliar with story-telling in current pedagogical structures, to be careful in implementing this method in their teaching.

Clonts, Jean G. (1992, January). The concept of reliability as it pertains to data from qualitative studies. Paper presented at the annual meeting of the Southwest Educational Research Association, Houston, TX.

Presents a review of literature on reliability in qualitative studies and defines reliability as the extent to which studies can be replicated by using the same methods and getting the same results. Strategies to enhance reliability through study design, data collection, and data analysis are suggested. Generalizability as an estimate of reliability is also explored.

Connelly, Michael F. & Clandinin D. Jean. (1990). Stories of experience and narrative inquiry. Educational Researcher, 19. (5), 2-14.

Describes narrative as a site of inquiry and a qualitative research methodology in which experiences of observer and observed interact. This form of research necessitates the development of new criteria, which may include apparency, verisimilitude, and transferability (7).

Crocker, Linda & Algina, James. (1986). Introduction to classical & modern test theory. New York: Holt, Rinehart and Winston.

Discusses test theory and its application to psychometrics. Chapters range from general overview of major issues to statistical methods and application.

Cronbach, Lee J. et al. (1967). The dependability of behavioral measurements: multifaceted studies of generalizability. Stanford: Stanford UP.

A technical research report that includes statistical methodology in order to contrast multifaceted generalizability with classical reliability.

Cziko, Gary A. (1992). Purposeful behavior as the control of perception: implications for educational research. Educational Researcher, 21 (9), 10-18. El-Hassan, Karma. (1995). Students' Rating of Instruction: Generalizability of Findings. Studies in Educational Research 21 (4), 411-29.

Issues of dimensionality, validity, reliability, and generalizability of students' ratings of instruction are discussed in relation to a study in which 610 college students who evaluated their instructors on the Teacher Effectiveness Scale.

Feingold, Alan. (1994). Gender differences in variability in intellectual abilities: a cross-cultural perspective. Sex Roles: A Journal of Research 20 (1-2), 81-93.

Feingold conducts a cross-cultural quantitative review of contemporary findings of gender differences in variability in verbal, mathematical, and spatial abilities to assess the generalizability of U.S. findings that males are more variable than females in mathematical and spatial abilities, and the sexes are equally variable in verbal ability.

Firestone,William A. (1993). Alternative arguments for generalizing from data as applied to qualitative research. Educational Researcher, 22 (4), 16-22.

Focuses on generalization in three areas of qualitative research: sample to population extrapolation, analytic generalization, and case-to-case transfer (16). Explains underlying principles, related theories, and criteria for each approach.

Fyans, Leslie J. (Ed.). (1983). Generalizability theory: Inferences and practical applications. In New Directions for Testing and Measurement: Vol. 18. San Francisco: Jossey-Bass.

A collection of articles on generalizability theory. The goal of the book is to present different aspects and applications of generalizability theory in a way that allows the reader to apply the theory.

Hammersley, Martyn. (Ed.). (1993). Social research: Philosophy, politics and practice. Newbury Park, CA: Sage Publications.

A collection of articles that provide an overview of positivism; includes an article on increasing the generalizability of qualitative research by Janet Ward Schofield.

Hathaway, R. (1995). Assumptions underlying quantitative and qualitative research: Implications for institutional research. Research in higher education, 36 (5), 535-562.

Hathaway says that the choice between using qualitative or quantitative approaches is less about methodology and more about aligning oneself with particular theoretical and academic traditions. He concluded that the two approaches address questions in very different ways, each one having its own advantages and drawbacks.

Heck, Ronald H., Marcoulides, George A. (1996). . Research in the Teaching of English 22 (1), 9-44.

Hipps, Jerome A. (1993). Trustworthiness and authenticity: Alternate ways to judge authentic assessments. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.

Contrasts the foundational assumptions of the constructivist approach to traditional research and the positivist approach to authentic assessment in relation to generalizability and other research issues.

Howe, Kenneth & Eisenhart, Margaret. (1990). Standards for qualitative (and quantitative) research: A prolegomenon. Educational Researcher, 19 (4), 2-9.

Huang, Chi-yu, et al. (1995, April). A generalizability theory approach to examining teaching evaluation instruments completed by students. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.

Presents the results of a study that used generalizability theory to investigate the reasons for variability in a teacher and course evaluation mechanism.

Hungerford, Harold R. et al. (1992). Investigating and Evaluating Environmental Issues and Actions: Skill Development Modules .

A guide designed to teach students how to investigate and evaluate environmental issues and actions. The guide is presented in six modules including information collecting and surveys, questionnaires, and opinionnaires.

Jackson, Philip W. (1990). The functions of educational research. Educational Researcher 19 (7), 3-9. Johnson, Randell G. (1993, April). A validity generalization study of the multiple assessment and program services test. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.

Presents results of study of validity reports of the Multiple Assessment and Program Services Test using quantitative analysis to determine the generalizability of the results.

Jones, Elizabeth A & Ratcliff, Gary. (1993). Critical thinking skills for college students. (National Center on Postsecondary Teaching, Learning, and Asessment). University Park, PA.

Reviews research literature exploring the nature of critical thinking; discusses the extent to which critical thinking is generalizable across disciplines.

Karpinski, Jakub. (1990). Causality in Sociological Research . Boston: Kluwer Academic Publishers.

Discusses causality and causal analysis in terms of sociological research. Provides equations and explanations.

Kirsch, Irwin S. & Jungeblut, Ann. (1995). Using large-scale assessment results to identify and evaluate generalizable indicators of literacy. (National Center on Adult Literacy, Publication No. TR94-19). Philadelphia, PA.

Reports analysis of data collected during an extensive literacy survey in order to help understand the different variables involved in literacy proficiency. Finds that literacy skills can be predicted across large, heterogeneous populations, but not as effectively across homogeneous populations.

Lauer, Janice M. & Asher, J. William. (1988). Composition research: empirical designs. New York: Oxford Press.

Explains the selection of subjects, formulation of hypotheses or questions, data collection, data analysis, and variable identification through discussion of each design.

LeCompte, Margaret & Goetz, Judith Preissle. (1982). Problems of reliability and validity in ethnographic research. Review of Educational Research, 52 (1), 31-60.

Concentrates on educational research and ethnography and shows how to better take reliability and validity into account when doing ethnographic research.

Marcoulides, George; Simkin, Mark G. (1991). Evaluating student papers: the case for peer review. Journal of Education for Business 67 (2), 80-83.

A preprinted evaluation form and generalizability theory are used to judge the reliability of student grading of their papers.

Maxwell, Joseph A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62 (3), 279-300.

Explores the five types of validity used in qualitative research, including generalizable validity, and examines possible threats to research validity.

McCarthy, Christine L. (1996, Spring). What is "critical thinking"? Is it generalizable? Educational Theory, 46 217-239.

Reviews, compares and contrasts a selection of essays from Stephen P. Norris' book The Generalizability of Critical Thinking: Multiple Perspectives on an Education Ideal in order to explore the diversity of the topic of critical thinking.

Miles, Matthew B. & Huberman, A. Michael. (1994). Qualitative data analysis. Thousand Oaks: Sage Publications.

A comprehensive review of data analysis. Subjects range from collecting data to producing an actual report.

Minium, Edward W. & King, M. Bruce, & Bear, Gordon. (1993). Statistical reasoning in psychology and education . New York: John Wiley & Sons, Inc.

A textbook designed to teach students about statistical data and theory.

Moss, Pamela A. (1992). Shifting conceptions of validity in educational measurement: Implications for performance assessment. Review of Educational Research, 62 (3), 229-258. Nachmias, David & Nachmias, Chava . (1981). Research methods in the social sciences. New York: St. Martin's Press.

Discusses the foundations of empirical research, data collection, data processing and analysis, inferential methods, and the ethics of social science research.

Nagy, Philip; Jarchow, Elaine McNally. (1981). Estimating variance components of essay ratings in a complex design. Speech/Conference Paper .

This paper discusses variables influencing written composition quality and how they can be best controlled to improve the reliability assessment of writing ability.

Nagy, William E., Herman, Patricia A., & Anderson, Richard C. (1985). Learning word meanings from context: How broadly generalizable? (University of Illinois at Urbana-Champaign. Center for the Study of Reading, Technical Report No. 347). Cambridge, MA: Bolt, Beranek and Newman.

Reports the results of a study that investigated how students learn word meanings while reading from context. Claims that the study was designed to be generalized.

Naizer, Gilbert. (1992, January). Basic concepts in generalizability theory: A more powerful approach to evaluating reliability. Presented at the annual meeting of the Southwest Educational Research Association, Houston, TX.

Discusses how a measurement approach called generalizability theory (G-theory) is an important alternative to the more classical measurement theory that yields less useful coefficients. G-theory is about the dependability of behavioral measurements that allows the simultaneous estimation of multiple sources of error variance.

Newman, Isadore & Macdonald, Suzanne. (1993, May). Interpreting qualitative data: A methodological inquiry. Paper presented at the annual meeting of the Ohio Academy of Science, Youngstown, OH.

Issues of consistency, triangulation, and generalizability are discussed in relation to a qualitative study involving graduate student participants. The authors refute Polkinghorne's views of the generalizability of qualitative research, arguing that quantitative research is more suitable for generalizability.

Norris, Stephen P. (Ed.). (1992). The generalizability of critical thinking: multiple perspectives on an education ideal. New York: Teachers College Press. A set of essays from a variety of disciplines presenting different perspectives on the topic of the generalizability of critical thinking. The authors refer and respond to each other. Peshkin, Alan. (1993). The goodness of qualitative research. Educational Researcher, 22 (2), 23-29.

Discusses how effective qualitative research can be in obtaining desired results and concludes that it is an important tool scholars can use in their explorations. The four categories of qualitative research--description, interpretation, verification, and evaluation--are examined.

Rafilson, Fred. (1991, July). The case for validity generalization.

Describes generalization as a quantitative process. Briefly discusses theory, method, examples, and applications of validity generalization, emphasizing unseen local methodological problems.

Rhodebeck, Laurie A. The structure of men's and women's feminist orientations: feminist identity and feminist opinion. Gender & Society 10 (4), 386-404.

This study considers two problems: the extent to which feminist opinions are distinct from feminist identity and the generalizability of these separate constructs across gender and time.

Runkel, Philip J. & McGrath, E. Joseph. (1972). Research on human behavior: A systematic guide to method. New York: Holt, Rinehart and Winston, Inc.

Discusses how researchers can utilize their experiences of human behavior and apply them to research in a systematic and explicit fashion.

Salomon, Gavriel. (1991). Transcending the qualitative-quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20 (6), 10-18.

Examines the complex issues/variables involved in studies. Two types of approaches are explored: an Analytic Approach, which assumes internal and external issues, and a Systematic Approach, in which each component affects the whole. Also discusses how a study can never fully measure how much x affects y because there are so many inter-relations. Knowledge is applied differently within each approach.

Schrag, Francis. (1992). In defense of positivist research paradigms. Educational Researcher, 21 (5), 5-8.

Positivist critics Elliot Eisner, Fredrick Erikson, Henry Giroux, and Thomas Popkewitz are logically committed to propositions that can be tested only by means of positivist research paradigms. A definition of positivism is gathered through example. Overall, it is concluded that educational research need not aspire to be practical.

Sekaran, Uma. (1984). Research methods for managers: A skill-building approach. New York: John Wiley and Sons.

Discusses managerial approaches to conducting research in organizations. Provides understandable definitions and explanations of such methods as sampling and data analysis and interpretation.

Shadish, William R. (1995). The logic of generalization: five principles common to experiments and ethnographies. American Journal of Community Psychology 23 (3), 419-29.

Both experiments and ethnographies are highly localized, so they are often criticized for lack of generalizability. This article describes a logic of generalization that may help solve such problems.

Shavelson, Richard J. & Webb, Noreen M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage Publications.

Snyder, I. (1995). Multiple perspectives in literacy research: Integrating the quantitative and qualitative. Language and Education, 9 (1), 45-59.

This article explains a study in which the author employed quantitative and qualitative methods simultaneously to compare computer composition classrooms and traditional classrooms. Although there were some problems with integrating both approaches, Snyder says they can be used together if researchers plan carefully and use their methods thoughtfully.

Stallings, William M. (1995). Confessions of a quantitative educational researcher trying to teach qualitative research. Educational Researcher, 24 (3), 31-32.

Discusses the trials and tribulations of teaching a qualitative research course to graduate students. The author describes the successes and failings he encounters and asks colleagues for suggestions of readings for his syllabus.

Wagner, Ellen D. (1993, January). Evaluating distance learning projects: An approach for cross-project comparisons. Paper presented at the annual meeting of the Association for educational Communication and Technology, New Orleans, LA.

Describes a methodology developed to evaluate distance learning projects in a way that takes into account specific institutional issues while producing generalizable, valid and reliable results that allow for discussion among different institutions.

Yin, Robert K. (1989). Case Study Research: Design and Methods. London: Sage Publications.

A small section on the application of generalizability in regards to case studies.

Barnes, Jeffrey,  Kerri Conrad, Christof Demont-Heinrich, Mary Graziano, Dawn Kowalski, Jamie Neufeld, Jen Zamora, & Mike Palmquist. (2005). Generalizability and Transferability. Writing@CSU . Colorado State University. https://writing.colostate.edu/guides/guide.cfm?guideid=65

External validity, generalizability, and knowledge utilization

Affiliation.

• 1 College of Nursing, University of Saskatchewan, 107 Wiggins Road, Saskatoon, Saskatchewan S7N 5E5, Canada. [email protected]
• PMID: 15098414
• DOI: 10.1111/j.1547-5069.2004.04006.x

Purpose: To examine the concepts of external validity and generalizability, and explore strategies to strengthen generalizability of research findings, because of increasing demands for knowledge utilization in an evidence-based practice environment.

Framework: The concepts of external validity and generalizability are examined, considering theoretical aspects of external validity and conflicting demands for internal validity in research designs. Methodological approaches for controlling threats to external validity and strategies to enhance external validity and generalizability of findings are discussed.

Conclusions: Generalizability of findings is not assured even if internal validity of a research study is addressed effectively through design. Strict controls to ensure internal validity can compromise generalizability. Researchers can and should use a variety of strategies to address issues of external validity and enhance generalizability of findings. Enhanced external validity and assessment of generalizability of findings can facilitate more appropriate use of research findings.

Publication types

• Data Collection / standards
• Data Interpretation, Statistical
• Diffusion of Innovation*
• Evidence-Based Medicine / standards
• Nursing Research / standards*
• Reproducibility of Results*
• Research Design / standards*

• Open access
• Published: 28 March 2024

Using the consolidated Framework for Implementation Research to integrate innovation recipients’ perspectives into the implementation of a digital version of the spinal cord injury health maintenance tool: a qualitative analysis

• John A Bourke 1 , 2 , 3 ,
• K. Anne Sinnott Jerram 1 , 2 ,
• Mohit Arora 1 , 2 ,
• Ashley Craig 1 , 2 &
• James W Middleton 1 , 2 , 4 , 5  

BMC Health Services Research volume  24 , Article number:  390 ( 2024 ) Cite this article

111 Accesses

Metrics details

Despite advances in managing secondary health complications after spinal cord injury (SCI), challenges remain in developing targeted community health strategies. In response, the SCI Health Maintenance Tool (SCI-HMT) was developed between 2018 and 2023 in NSW, Australia to support people with SCI and their general practitioners (GPs) to promote better community self-management. Successful implementation of innovations such as the SCI-HMT are determined by a range of contextual factors, including the perspectives of the innovation recipients for whom the innovation is intended to benefit, who are rarely included in the implementation process. During the digitizing of the booklet version of the SCI-HMT into a website and App, we used the Consolidated Framework for Implementation Research (CFIR) as a tool to guide collection and analysis of qualitative data from a range of innovation recipients to promote equity and to inform actionable findings designed to improve the implementation of the SCI-HMT.

Data from twenty-three innovation recipients in the development phase of the SCI-HMT were coded to the five CFIR domains to inform a semi-structured interview guide. This interview guide was used to prospectively explore the barriers and facilitators to planned implementation of the digital SCI-HMT with six health professionals and four people with SCI. A team including researchers and innovation recipients then interpreted these data to produce a reflective statement matched to each domain. Each reflective statement prefaced an actionable finding, defined as alterations that can be made to a program to improve its adoption into practice.

Five reflective statements synthesizing all participant data and linked to an actionable finding to improve the implementation plan were created. Using the CFIR to guide our research emphasized how partnership is the key theme connecting all implementation facilitators, for example ensuring that the tone, scope, content and presentation of the SCI-HMT balanced the needs of innovation recipients alongside the provision of evidence-based clinical information.

Conclusions

Understanding recipient perspectives is an essential contextual factor to consider when developing implementation strategies for healthcare innovations. The revised CFIR provided an effective, systematic method to understand, integrate and value recipient perspectives in the development of an implementation strategy for the SCI-HMT.

Trial registration

Peer Review reports

Injury to the spinal cord can occur through traumatic causes (e.g., falls or motor vehicle accidents) or from non-traumatic disease or disorder (e.g., tumours or infections) [ 1 ]. The onset of a spinal cord injury (SCI) is often sudden, yet the consequences are lifelong. The impact of a SCI is devastating, with effects on sensory and motor function, bladder and bowel function, sexual function, level of independence, community participation and quality of life [ 2 ]. In order to maintain good health, wellbeing and productivity in society, people with SCI must develop self-management skills and behaviours to manage their newly acquired chronic health condition [ 3 ]. Given the increasing emphasis on primary health care and community management of chronic health conditions, like SCI, there is a growing responsibility on all parties to promote good health practices and minimize the risks of common health complications in their communities.

To address this need, the Spinal Cord Injury Health Maintenance Tool (SCI-HMT) was co-designed between 2018 and 2023 with people living with SCI and their General Practitioners (GPs) in NSW, Australia [ 4 ] The aim of the SCI-HMT is to support self-management of the most common and arguably avoidable potentially life-threatening complications associated with SCI, such as mental health crises, autonomic dysreflexia, kidney infections and pressure injuries. The SCI-HMT provides comprehensible information with resources about the six highest priority health areas related to SCI (as indicated by people with SCI and GPs) and was developed over two phases. Phase 1 focused on developing a booklet version and Phase 2 focused on digitizing this content into a website and smartphone app [ 4 , 5 ].

Enabling the successful implementation of evidence-based innovations such as the SCI-HMT is inevitably influenced by contextual factors: those dynamic and diverse array of forces within real-world settings working for or against implementation efforts [ 6 ]. Contextual factors often include background environmental elements in which an intervention is situated, for example (but not limited to) demographics, clinical environments, organisational culture, legislation, and cultural norms [ 7 ]. Understanding the wider context is necessary to identify and potentially mitigate various challenges to the successful implementation of those innovations. Such work is the focus of determinant frameworks, which focus on categorising or classing groups of contextual determinants that are thought to predict or demonstrate an effect on implementation effectiveness to better understand factors that might influence implementation outcomes [ 8 ].

One of the most highly cited determinant frameworks is the Consolidated Framework for Implementation Research (CFIR) [ 9 ], which is often posited as an ideal framework for pre-implementation preparation. Originally published in 2009, the CFIR has recently been subject to an update by its original authors, which included a literature review, survey of users, and the creation of an outcome addendum [ 10 , 11 ]. A key contribution from this revision was the need for a greater focus on the place of innovation recipients, defined as the constituency for whom the innovation is being designed to benefit; for example, patients receiving treatment, students receiving a learning activity. Traditionally, innovation recipients are rarely positioned as key decision-makers or innovation implementers [ 8 ], and as a consequence, have not often been included in the application of research using frameworks, such as the CFIR [ 11 ].

Such power imbalances within the intersection of healthcare and research, particularly between those receiving and delivering such services and those designing such services, have been widely reported [ 12 , 13 ]. There are concerted efforts within health service development, health research and health research funding, to rectify this power imbalance [ 14 , 15 ]. Importantly, such efforts to promote increased equitable population impact are now being explicitly discussed within the implementation science literature. For example, Damschroder et al. [ 11 ] has recently argued for researchers to use the CFIR to collect data from innovation recipients, and that, ultimately, “equitable population impact is only possible when recipients are integrally involved in implementation and all key constituencies share power and make decisions together” (p. 7). Indeed, increased equity between key constituencies and partnering with innovation recipients promotes the likelihood of sustainable adoption of an innovation [ 4 , 12 , 14 ].

There is a paucity of work using the updated CFIR to include and understand innovation recipients’ perspectives. To address this gap, this paper reports on a process of using the CFIR to guide the collection of qualitative data from a range of innovation recipients within a wider co-design mixed methods study examining the development and implementation of SCI-HMT. The innovation recipients in our research are people living with SCI and GPs. Guided by the CFIR domains (shown in the supplementary material), we used reflexive thematic analysis [ 16 ]to summarize data into reflective summaries, which served to inform actionable findings designed to improve implementation of the SCI-HMT.

The procedure for this research is multi-stepped and is summarized in Fig.  1 . First, we mapped retrospective qualitative data collected during the development of the SCI-HMT [ 4 ] against the five domains of the CFIR in order to create a semi-structured interview guide (Step 1). Then, we used this interview guide to collect prospective data from health professionals and people with SCI during the development of the digital version of the SCI-HMT (Step 2) to identify implementation barriers and facilitators. This enabled us to interpret a reflective summary statement for each CFIR domain. Lastly, we developed an actionable finding for each domain summary. The first (RESP/18/212) and second phase (2019/ETH13961) of the project received ethical approval from The Northern Sydney Local Health District Human Research Ethics Committee. The reporting of this study was conducted in line with the consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines [ 17 ]. All methods were performed in accordance with the relevant guidelines and regulations.

Procedure of synthesising datasets to inform reflective statements and actionable findings. a Two health professionals had a SCI (one being JAB); b Two co-design researchers had a SCI (one being JAB)

Step one: retrospective data collection and analysis

We began by retrospectively analyzing the data set (interview and focus group transcripts) from the previously reported qualitative study from the development phase of the SCI-HMT [ 4 ]. This analysis was undertaken by two team members (KASJ and MA). KASJ has a background in co-design research. Transcript data were uploaded into NVivo software (Version 12: QSR International Pty Ltd) and a directed content analysis approach [ 18 ] was applied to analyze categorized data a priori according to the original 2009 CFIR domains (intervention characteristics, outer setting, inner setting, characteristics of individuals, and process of implementation) described by Damschroder et al. [ 9 ]. This categorized data were summarized and informed the specific questions of a semi-structured interview guide. The final output of step one was an interview guide with context-specific questions arranged according to the CFIR domains (see supplementary file 1). The interview was tested with two people with SCI and one health professional.

Step two: prospective data collection and analysis

In the second step, semi-structured interviews were conducted by KASJ (with MA as observer) with consenting healthcare professionals who had previously contributed to the development of the SCI-HMT. Healthcare professionals included GPs, Nurse Consultants, Specialist Physiotherapists, along with Health Researchers (one being JAB). In addition, a focus group was conducted with consenting individuals with SCI who had contributed to the SCI-HMT design and development phase. The interview schedule designed in step one above guided data collection in all interviews and the focus group.

The focus group and interviews were conducted online, audio recorded, transcribed verbatim and uploaded to NVivo software (Version 12: QSR International Pty Ltd). All data were subject to reflexive, inductive and deductive thematic analysis [ 16 , 19 ] to better understand participants’ perspectives regarding the potential implementation of the SCI-HMT. First, one team member (KASJ) read transcripts and began a deductive analysis whereby data were organized into CFIR domains-specific dataset. Second, KASJ and JAB analyzed this domain-specific dataset to inductively interpret a reflective statement which served to summarise all participant responses to each domain. The final output of step two was a reflective summary statement for each CFIR domain.

Step three: data synthesis

In the third step we aimed to co-create an actionable finding (defined as tangible alteration that can be made to a program, in this case the SCI-HMT [ 20 ]) based on each domain-specific reflective statement. To achieve this, three codesign researchers (KAS and JAB with one person with SCI from Step 2 (deidentified)) focused on operationalising each reflective statement into a recommended modification for the digital version of the SCI-HMT. This was an iterative process guided by the specific CFIR domain and construct definitions, which we deemed salient and relevant to each reflective statement (see Table  2 for example). Data synthesis involved line by line analysis, group discussion, and repeated refinement of actionable findings. A draft synthesis was shared with SCI-HMT developers (JWM and MA) and refinement continued until consensus was agreed on. The final outputs of step three were an actionable finding related to each reflective statement for each CFIR domain.

The characteristics of both the retrospective and prospective study participants are shown in Table  1 . The retrospective data included data from a total of 23 people: 19 people with SCI and four GPs. Of the 19 people with SCI, 12 participated in semi-structured interviews, seven participated in the first focus group, and four returned to the second focus group. In step 2, four people with SCI participated in a focus group and six healthcare professionals participated in one-on-one semi-structured interviews. Two of the healthcare professionals (a GP and a registrar) had lived experience of SCI, as did one researcher (JAB). All interviews and focus groups were conducted either online or in-person and ranged in length between 60 and 120 min.

In our overall synthesis, we actively interpreted five reflective statements based on the updated CFIR domain and construct definitions by Damschroder et al. [ 11 ]. Table  2 provides a summary of how we linked the updated CFIR domain and construct definitions to the reflective statements. We demonstrate this process of co-creation below, including illustrative quotes from participants. Importantly, we guide readers to the actionable findings related to each reflective statement in Table  2 . Each actionable statement represents an alteration that can be made to a program to improve its adoption into practice.

Participants acknowledged that self-management is a major undertaking and very demanding, as one person with SCI said, “ we need to be informed without being terrified and overwhelmed”. Participants felt the HMT could indeed be adapted, tailored, refined, or reinvented to meet local needs. For example, another person with SCI remarked:

“Education needs to be from the get-go but in bite sized pieces from all quarters when readiness is most apparent… at all time points , [not just as a] a newbie tool or for people with [long-term impairment] ” (person with SCI_02).

Therefore, the SCI-HMT had to balance complexity of content while still being accessible and engaging, and required input from both experts in the field and those with lived experience of SCI, for example, a clinical nurse specialist suggested:

“it’s essential [the SCI-HMT] is written by experts in the field as well as with collaboration with people who have had a, you know, the lived experience of SCI” (healthcare professional_03).

Furthermore, the points of contact with healthcare for a person with SCI can be challenging to navigate and the SCI-HMT has the potential to facilitate a smoother engagement process and improve communication between people with SCI and healthcare services. As a GP suggested:

“we need a tool like this to link to that pathway model in primary health care , [the SCI-HMT] it’s a great tool, something that everyone can read and everyone’s reading the same thing” (healthcare professional_05).

Participants highlighted that the ability of the SCI-HMT to facilitate effective communication was very much dependent on the delivery format. The idea of digitizing the SCI-HMT garnered equal support from people with SCI and health care professionals, with one participant with SCI deeming it to be “ essential” ( person with SCI_01) and a health professional suggesting a “digitalized version will be an advantage for most people” (healthcare professional_02).

Outer setting

There was strong interest expressed by both people with SCI and healthcare professionals in using the SCI-HMT. The fundamental premise was that knowledge is power and the SCI-HMT would have strong utility in post-acute rehabilitation services, as well as primary care. As a person with SCI said,

“ we need to leave the [spinal unit] to return to the community with sufficient knowledge, and to know the value of that knowledge and then need to ensure primary healthcare provider [s] are best informed” (person with SCI_04).

The value of the SCI-HMT in facilitating clear and effective communication and shared decision-making between healthcare professionals and people with SCI was also highlighted, as shown by the remarks of an acute nurse specialist:

“I think this tool is really helpful for the consumer and the GP to work together to prioritize particular tests that a patient might need and what the regularity of that is” (healthcare professional_03).

Engaging with SCI peer support networks to promote the SCI-HMT was considered crucial, as one person with SCI emphasized when asked how the SCI-HMT might be best executed in the community, “…peers, peers and peers” (person with SCI_01). Furthermore, the layering of content made possible in the digitalized version will allow for the issue of approachability in terms of readiness for change, as another person with SCI said:

“[putting content into a digital format] is essential and required and there is a need to put summarized content in an App with links to further web-based information… it’s not likely to be accessed otherwise” (person with SCI_02).

Inner setting

Participants acknowledged that self-management of health and well-being is substantial and demanding. It was suggested that the scope, tone, and complexity of the SCI-HMT, while necessary, could potentially be resisted by people with SCI if they felt overwhelmed, as one person with SCI described:

“a manual that is really long and wordy, like, it’s [a] health metric… they maybe lack the health literacy to, to consume the content then yes, it would impede their readiness for [self-management]” (person with SCI_02).

Having support from their GPs was considered essential, and the HMT could enable GP’s, who are under time pressure, to provide more effective health and advice to their patients, as one GP said:

“We GP’s are time poor, if you realize then when you’re time poor you look quickly to say oh this is a patient tool - how can I best use this?” (healthcare professional_05).

Furthermore, health professional skills may be best used with the synthesis of self-reported symptoms, behaviors, or observations. A particular strength of a digitized version would be its ability to facilitate more streamlined communication between a person with SCI and their primary healthcare providers developing healthcare plans, as an acute nurse specialist reflected, “ I think that a digitalized version is essential with links to primary healthcare plans” (healthcare professional_03).

Efficient communication with thorough assessment is essential to ensure serious health issues are not missed, as findings reinforce that the SCI-HMT is an educational tool, not a replacement for healthcare services, as a clinical nurse specialist commented, “ remember, things will go wrong– people end up very sick and in acute care “ (healthcare professional_02).

The SCI-HMT has the potential to provide a pathway to a ‘hope for better than now’ , a hope to ‘remain well’ and a hope to ‘be happy’ , as the informant with SCI (04) declared, “self-management is a long game, if you’re keeping well, you’ve got that possibility of a good life… of happiness”. Participants with SCI felt the tool needed to be genuine and

“acknowledge the huge amount of adjustment required, recognizing that dealing with SCI issues is required to survive and live a good life” (person with SCI_04).

However, there is a risk that an individual is completely overwhelmed by the scale of the SCI-HMT content and the requirement for lifelong vigilance. Careful attention and planning were paid to layering the information accordingly to support self-management as a ‘long game’, which one person with SCI reflected in following:

“the first 2–3 year [period] is probably the toughest to get your head around the learning stuff, because you’ve got to a stage where you’re levelling out, and you’ve kind of made these promises to yourself and then you realize that there’s no quick fix” (person with SCI_01).

It was decided that this could be achieved by providing concrete examples and anecdotes from people with SCI illustrating that a meaningful, healthy life is possible, and that good health is the bedrock of a good life with SCI.

There was universal agreement that the SCI-HMT is aspirational and that it has the potential to improve knowledge and understanding for people with SCI, their families, community workers/carers and primary healthcare professionals, as a GP remarked:

“[different groups] could just read it and realize, ‘Ahh, OK that’s what that means… when you’re doing catheters. That’s what you mean when you’re talking about bladder and bowel function or skin care” (healthcare professional_04).

Despite the SCI-HMT providing an abundance of information and resources to support self-management, participants identified four gaps: (i) the priority issue of sexuality, including pleasure and identity, as one person with SCI remarked:

“ sexuality is one of the biggest issues that people with SCI often might not speak about that often cause you know it’s awkward for them. So yeah, I think that’s a that’s a serious issue” (person with SCI_03).

(ii) consideration of the taboo nature of bladder and bowel topics for indigenous people, (iii) urgent need to ensure links for SCI-HMT care plans are compatible with patient management systems, and (iv) exercise and leisure as a standalone topic taking account of effects of physical activity, including impact on mental health and wellbeing but more especially for fun.

To ensure longevity of the SCI-HMT, maintaining a partnership between people with SCI, SCI community groups and both primary and tertiary health services is required for liaison with the relevant professional bodies, care agencies, funders, policy makers and tertiary care settings to ensure ongoing education and promotion of SCI-HMT is maintained. For example, delivery of ongoing training of healthcare professionals to both increase the knowledge base of primary healthcare providers in relation to SCI, and to promote use of the tools and resources through health communities. As a community nurse specialist suggested:

“ improving knowledge in the health community… would require digital links to clinical/health management platforms” (healthcare professional_02).

In a similar vein, a GP suggested:

“ our common GP body would have continuing education requirements… especially if it’s online, in particular for the rural, rural doctors who you know, might find it hard to get into the city” (healthcare professional_04).

The successful implementation of evidence-based innovations into practice is dependent on a wide array of dynamic and active contextual factors, including the perspectives of the recipients who are destined to use such innovations. Indeed, the recently updated CFIR has called for innovation recipient perspectives to be a priority when considering contextual factors [ 10 , 11 ]. Understanding and including the perspectives of those the innovation is being designed to benefit can promote increased equity and validation of recipient populations, and potentially increase the adoption and sustainability of innovations.

In this paper, we have presented research using the recently updated CFIR to guide the collection of innovation recipients’ perspectives (including people with SCI and GPs working in the community) regarding the potential implementation barriers and facilitators of the digital version of the SCI-HMT. Collected data were synthesized to inform actionable findings– tangible ways in which the SCI-HMT could be modified according of the domains of the CFIR (e.g., see Keith et al. [ 20 ]). It is important to note that we conducted this research using the original domains of the CFIR [ 9 ] prior to Damschroder et al. publishing the updated CFIR [ 11 ]. However, in our analysis we were able to align our findings to the revised CFIR domains and constructs, as Damschroder [ 11 ] suggests, constructs can “be mapped back to the original CFIR to ensure longitudinal consistency” (p. 13).

One of the most poignant findings from our analyses was the need to ensure the content of the SCI-HMT balanced scientific evidence and clinical expertise with lived experience knowledge. This balance of clinical and experiential knowledge demonstrated genuine regard for lived experience knowledge, and created a more accessible, engaging, useable platform. For example, in the innovation and individual domains, the need to include lived experience quotes was immediately apparent once the perspective of people with SCI was included. It was highlighted that while the SCI-HMT will prove useful to many parties at various stages along the continuum of care following onset of SCI, there will be those individuals that are overwhelmed by the scale of the content. That said, the layering of information facilitated by the digitalized version is intended to provide an ease of navigation through the SCI-HMT and enable a far greater sense of control over personal health and wellbeing. Further, despite concerns regarding e-literacy the digitalized version of the SCI-HMT is seen as imperative for accessibility given the wide geographic diversity and recent COVID pandemic [ 21 ]. While there will be people who are challenged by the technology, the universally acceptable use of the internet is seen as less of a barrier than printed material.

The concept of partnership was also apparent within the data analysis focusing on the outer and inner setting domains. In the outer setting domain, our findings emphasized the importance of engaging with SCI community groups, as well as primary and tertiary care providers to maximize uptake at all points in time from the phase of subacute rehabilitation onwards. While the SCI-HMT is intended for use across the continuum of care from post-acute rehabilitation onwards, it may be that certain modules are more relevant at different times, and could serve as key resources during the hand over between acute care, inpatient rehabilitation and community reintegration.

Likewise, findings regarding the inner setting highlighted the necessity of a productive partnership between GPs and individuals with SCI to address the substantial demands of long-term self-management of health and well-being following SCI. Indeed, support is crucial, especially when self-management is the focus. This is particularly so in individuals living with complex disability following survival after illness or injury [ 22 ], where health literacy has been found to be a primary determinant of successful health and wellbeing outcomes [ 23 ]. For people with SCI, this tool potentially holds the most appeal when an individual is ready and has strong partnerships and supportive communication. This can enable potential red flags to be recognized earlier allowing timely intervention to avert health crises, promoting individual well-being, and reducing unnecessary demands on health services.

While the SCI-HMT is an educational tool and not meant to replace health services, findings suggest the current structure would lead nicely to having the conversation with a range of likely support people, including SCI peers, friends and family, GP, community nurses, carers or via on-line support services. The findings within the process domain underscored the importance of ongoing partnership between innovation implementers and a broad array of innovation recipients (e.g., individuals with SCI, healthcare professionals, family, funding agencies and policy-makers). This emphasis on partnership also addresses recent discussions regarding equity and the CFIR. For example, Damschroder et al. [ 11 ] suggests that innovation recipients are too often not included in the CFIR process, as the CFIR is primarily seen as a tool intended “to collect data from individuals who have power and/or influence over implementation outcomes” (p. 5).

Finally, we feel that our inclusion of innovation recipients’ perspectives presented in this article begins to address the notion of equity in implementation, whereby the inclusion of recipient perspectives in research using the CFIR both validates, and increases, the likelihood of sustainable adoption of evidence-based innovations, such as the SCI-HMT. We have used the CFIR in a pragmatic way with an emphasis on meaningful engagement between the innovation recipients and the research team, heeding the call from Damschroder et al. [ 11 ], who recently argued for researchers to use the CFIR to collect data from innovation recipients. Adopting this approach enabled us to give voice to innovation recipient perspectives and subsequently ensure that the tone, scope, content and presentation of the SCI-HMT balanced the needs of innovation recipients alongside the provision of evidence-based clinical information.

Our research is not without limitations. While our study was successful in identifying a number of potential barriers and facilitators to the implementation of the SCI-HMT, we did not test any implementation strategies to impact determinants, mechanisms, or outcomes. This will be the focus of future research on this project, which will investigate the impact of implementation strategies on outcomes. Focus will be given to the context-mechanism configurations which give rise to particular outcomes for different groups in certain circumstances [ 7 , 24 ]. A second potential concern is the relatively small sample size of participants that may not allow for saturation and generalizability of the findings. However, both the significant impact of secondary health complications for people with SCI and the desire for a health maintenance tool have been established in Australia [ 2 , 4 ]. The aim our study reported in this article was to achieve context-specific knowledge of a small sample that shares a particular mutual experience and represents a perspective, rather than a population [ 25 , 26 ]. We feel our findings can stimulate discussion and debate regarding participant-informed approaches to implementation of the SCI-HMT, which can then be subject to larger-sample studies to determine their generalisability, that is, their external validity. Notably, future research could examine the interaction between certain demographic differences (e.g., gender) of people with SCI and potential barriers and facilitators to the implementation of the SCI-HMT. Future research could also include the perspectives of other allied health professionals working in the community, such as occupational therapists. Lastly, while our research gave significant priority to recipient viewpoints, research in this space would benefit for ensuring innovation recipients are engaged as genuine partners throughout the entire research process from conceptualization to implementation.

Employing the CFIR provided an effective, systematic method for identifying recipient perspectives regarding the implementation of a digital health maintenance tool for people living with SCI. Findings emphasized the need to balance clinical and lived experience perspectives when designing an implementation strategy and facilitating strong partnerships with necessary stakeholders to maximise the uptake of SCI-HMT into practice. Ongoing testing will monitor the uptake and implementation of this innovation, specifically focusing on how the SCI-HMT works for different users, in different contexts, at different stages and times of the rehabilitation journey.

Data availability

The datasets supporting the conclusions of this article are available available upon request and with permission gained from the project Steering Committee.

Abbreviations

spinal cord injury

HMT-Spinal Cord Injury Health Maintenance Tool

Consolidated Framework for Implementation Research

Kirshblum S, Vernon WL. Spinal Cord Medicine, Third Edition. New York: Springer Publishing Company; 2018.

Middleton JW, Arora M, Kifley A, Clark J, Borg SJ, Tran Y, et al. Australian arm of the International spinal cord Injury (Aus-InSCI) Community Survey: 2. Understanding the lived experience in people with spinal cord injury. Spinal Cord. 2022;60(12):1069–79.

Article   PubMed   PubMed Central   Google Scholar  

Craig A, Nicholson Perry K, Guest R, Tran Y, Middleton J. Adjustment following chronic spinal cord injury: determining factors that contribute to social participation. Br J Health Psychol. 2015;20(4):807–23.

Article   PubMed   Google Scholar  

Middleton JW, Arora M, Jerram KAS, Bourke J, McCormick M, O’Leary D, et al. Co-design of the Spinal Cord Injury Health Maintenance Tool to support Self-Management: a mixed-methods Approach. Top Spinal Cord Injury Rehabilitation. 2024;30(1):59–73.

Article   Google Scholar  

Middleton JW, Arora M, McCormick M, O’Leary D. Health maintenance Tool: how to stay healthy and well with a spinal cord injury. A tool for consumers by consumers. 1st ed. Sydney, NSW Australia: Royal Rehab and The University of Sydney; 2020.

Google Scholar  

Nilsen P, Bernhardsson S. Context matters in implementation science: a scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Serv Res. 2019;19(1):189.

Jagosh J. Realist synthesis for Public Health: building an Ontologically Deep understanding of how Programs Work, for whom, and in which contexts. Annu Rev Public Health. 2019;40(1):361–72.

Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):53.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.

Damschroder LJ, Reardon CM, Opra Widerquist MA, Lowery JC. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR outcomes Addendum. Implement Sci. 2022;17(1):7.

Damschroder LJ, Reardon CM, Widerquist MAO, Lowery JC. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.

Plamondon K, Ndumbe-Eyoh S, Shahram S. 2.2 Equity, Power, and Transformative Research Coproduction. Research Co-Production in Healthcare2022. p. 34–53.

Verville L, Cancelliere C, Connell G, Lee J, Munce S, Mior S, et al. Exploring clinicians’ experiences and perceptions of end-user roles in knowledge development: a qualitative study. BMC Health Serv Res. 2021;21(1):926.

Gainforth HL, Hoekstra F, McKay R, McBride CB, Sweet SN, Martin Ginis KA, et al. Integrated Knowledge Translation Guiding principles for conducting and Disseminating Spinal Cord Injury Research in Partnership. Arch Phys Med Rehabil. 2021;102(4):656–63.

Langley J, Knowles SE, Ward V. Conducting a Research Coproduction Project. Research Co-Production in Healthcare2022. p. 112– 28.

Braun V, Clarke V. One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology. 2020:1–25.

Tong A, Sainsbury p, Craig J. Consolidated criteria for reporting qulaitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Bengtsson M. How to plan and perform a qualitative study using content analysis. NursingPlus Open. 2016;2:8–14.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77–101.

Keith RE, Crosson JC, O’Malley AS, Cromp D, Taylor EF. Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implement Science: IS. 2017;12(1):15.

Choukou M-A, Sanchez-Ramirez DC, Pol M, Uddin M, Monnin C, Syed-Abdul S. COVID-19 infodemic and digital health literacy in vulnerable populations: a scoping review. Digit HEALTH. 2022;8:20552076221076927.

PubMed   PubMed Central   Google Scholar  

Daniels N. Just Health: Meeting Health needs fairly. Cambridge University Press; 2007. p. 397.

Parker SM, Stocks N, Nutbeam D, Thomas L, Denney-Wilson E, Zwar N, et al. Preventing chronic disease in patients with low health literacy using eHealth and teamwork in primary healthcare: protocol for a cluster randomised controlled trial. BMJ Open. 2018;8(6):e023239–e.

Salter KL, Kothari A. Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci. 2014;9(1):115.

Sebele-Mpofu FY. The Sampling Conundrum in qualitative research: can Saturation help alleviate the controversy and alleged subjectivity in Sampling? Int’l J Soc Sci Stud. 2021;9:11.

Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by Information Power. Qual Health Res. 2015;26(13):1753–60.

Download references

Acknowledgements

Authors of this study would like to thank all the consumers with SCI and healthcare professionals for their invaluable contribution to this project. Their participation and insights have been instrumental in shaping the development of the SCI-HMT. The team also acknowledges the support and guidance provided by the members of the Project Steering Committee, as well as the partner organisations, including NSW Agency for Clinical Innovation, and icare NSW. Author would also like to acknowledge the informant group with lived experience, whose perspectives have enriched our understanding and informed the development of SCI-HMT.

The SCI Wellness project was a collaborative project between John Walsh Centre for Rehabilitation Research at The University of Sydney and Royal Rehab. Both organizations provided in-kind support to the project. Additionally, the University of Sydney and Royal Rehab received research funding from Insurance and Care NSW (icare NSW) to undertake the SCI Wellness Project. icare NSW do not take direct responsibility for any of the following: study design, data collection, drafting of the manuscript, or decision to publish.

Author information

Authors and affiliations.

John Walsh Centre for Rehabilitation Research, Northern Sydney Local Health District, St Leonards, NSW, Australia

John A Bourke, K. Anne Sinnott Jerram, Mohit Arora, Ashley Craig & James W Middleton

The Kolling Institute, Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, Australia

Burwood Academy Trust, Burwood Hospital, Christchurch, New Zealand

John A Bourke

Royal Rehab, Ryde, NSW, Australia

James W Middleton

State Spinal Cord Injury Service, NSW Agency for Clinical Innovation, St Leonards, NSW, Australia

You can also search for this author in PubMed   Google Scholar

Contributions

Project conceptualization: KASJ, MA, JWM; project methodology: JWM, MA, KASJ, JAB; data collection: KASJ and MA; data analysis: KASJ, JAB, MA, JWM; writing—original draft preparation: JAB; writing—review and editing: JAB, KASJ, JWM, MA, AC; funding acquisition: JWM, MA. All authors contributed to the revision of the paper and approved the final submitted version.

Corresponding author

Correspondence to John A Bourke .

Ethics declarations

Ethics approval and consent to participate.

The first (RESP/18/212) and second phase (2019/ETH13961) of the project received ethical approval from The Northern Sydney Local Health District Human Research Ethics Committee. All participants provided informed, written consent. All data were to be retained for 7 years (23rd May 2030).

Consent for publication

Not applicable.

Competing interests

MA part salary (from Dec 2018 to Dec 2023), KASJ part salary (July 2021 to Dec 2023) and JAB part salary (Jan 2022 to Aug 2022) was paid from the grant monies. Other authors declare no conflicts of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Bourke, J.A., Jerram, K.A.S., Arora, M. et al. Using the consolidated Framework for Implementation Research to integrate innovation recipients’ perspectives into the implementation of a digital version of the spinal cord injury health maintenance tool: a qualitative analysis. BMC Health Serv Res 24 , 390 (2024). https://doi.org/10.1186/s12913-024-10847-x

Download citation

Received : 14 August 2023

Accepted : 11 March 2024

Published : 28 March 2024

DOI : https://doi.org/10.1186/s12913-024-10847-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Spinal Cord injury
• Self-management
• Innovation recipients
• Secondary health conditions
• Primary health care
• Evidence-based innovations
• Actionable findings
• Consolidated Framework for implementation research

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 02 April 2024

A framework for the evaluation and reporting of incidental findings in clinical genomic testing

• Carolyn M. Brown   ORCID: orcid.org/0000-0003-4587-0002 1 ,
• Laura M. Amendola 1   na1 ,
• Anjana Chandrasekhar 1   na1 ,
• R. Tanner Hagelstrom 2 ,
• Gillian Halter 3 ,
• Akanchha Kesari 1 ,
• Erin Thorpe 1 ,
• Denise L. Perry 1 ,
• Ryan J. Taft 1 &
• Alison J. Coffey   ORCID: orcid.org/0000-0002-1338-8513 1  

European Journal of Human Genetics ( 2024 ) Cite this article

1 Altmetric

Metrics details

• Genetic testing
• Genetics research

Currently, there are no widely accepted recommendations in the genomics field guiding the return of incidental findings (IFs), defined here as unexpected results that are unrelated to the indication for testing. Consequently, reporting policies for IFs among laboratories offering genomic testing are variable and may lack transparency. Herein we describe a framework developed to guide the evaluation and return of IFs encountered in probands undergoing clinical genome sequencing (cGS). The framework prioritizes clinical significance and actionability of IFs and follows a stepwise approach with stopping points at which IFs may be recommended for return or not. Over 18 months, implementation of the framework in a clinical laboratory facilitated the return of actionable IFs in 37 of 720 (5.1%) individuals referred for cGS, which is reduced to 3.1% if glucose-6-phosphate dehydrogenase (G6PD) deficiency is excluded. This framework can serve as a model to standardize reporting of IFs identified during genomic testing.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 12 print issues and online access

251,40 € per year

only 20,95 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Randomized prospective evaluation of genome sequencing versus standard-of-care as a first molecular diagnostic test

Deanna G. Brockman, Christina A. Austin-Tse, … Miriam S. Udler

Recommendations for reporting results of diagnostic genomic testing

Zandra C. Deans, Joo Wook Ahn, … Christi J. van Asperen

Opportunistic genomic screening. Recommendations of the European Society of Human Genetics

Guido de Wert, Wybo Dondorp, … On behalf of the European Society of Human Genetics

Data availability

Gene–disease validity classifications are available through ClinGen [ 19 ] and/or the GenCC [ 20 ]. Variant classifications are available through ClinVar [ 39 ]. Actionability curations are available through ClinGen [ 19 ]. Further information and requests should be directed to and will be fulfilled by the lead contact, CMB ([email protected]) or senior author AJC ([email protected]).

Shkedi-Rafid S, Dheensa S, Crawford G, Fenwick A, Lucassen A. Defining and managing incidental findings in genetic and genomic practice. J Med Genet. 2014;51:715–23.

Article   PubMed   Google Scholar  

Ackerman SL, Koenig BA. Understanding variations in secondary findings reporting practices across U.S. genome sequencing laboratories. AJOB Empir Bioeth. 2018;9:48–57.

Beshir L. A framework to ethically approach incidental findings in genetic research. EJIFCC. 2020;31:302–9.

CAS   PubMed   PubMed Central   Google Scholar  

Vears D, Amor DJ. A framework for reporting secondary and incidental findings in prenatal sequencing: when and for whom? Prenat Diagn. 2022;42:697–704.

Article   PubMed   PubMed Central   Google Scholar  

Bowling KM, Thompson ML, Kelly MA, Scollon S, Slavotinek AM, Powell BC, et al. Return of non-ACMG recommended incidental genetic findings to pediatric patients: considerations and opportunities from experiences in genomic sequencing. Genome Med. 2022;14:131.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Fernandez CV, Bouffet E, Malkin D, Jabado N, O’Connell C, Avard D, et al. Attitudes of parents toward the return of targeted and incidental genomic research findings in children. Genet Med. 2014;16:633–40.

Loud JT, Bremer RC, Mai PL, Peters JA, Giri N, Stewart DR, et al. Research participant interest in primary, secondary, and incidental genomic findings. Genet Med. 2016;18:1218–25.

Sundby A, Boolsen MW, Burgdorf KS, Ullum H, Hansen TF, Middleton A, et al. Stakeholders in psychiatry and their attitudes toward receiving pertinent and incident findings in genomic research. Am J Med Genet A. 2017;173:2649–58.

AlFayyad I, Al-Tannir M, Abu-Shaheen A, AlGhamdi S. To disclose, or not to disclose? Perspectives of clinical genomics professionals toward returning incidental findings from genomic research. BMC Med Ethics. 2021;22:101.

de Wert G, Dondorp W, Clarke A, Dequeker EMC, Cordier C, Deans Z, et al. Opportunistic genomic screening. Recommendations of the European Society of Human Genetics. Eur J Hum Genet. 2021;29:365–77.

Boycott K, Hartley T, Adam S, Bernier F, Chong K, Fernandez BA, et al. The clinical application of genome-wide sequencing for monogenic diseases in Canada: position statement of the Canadian College of Medical Geneticists. J Med Genet. 2015;52:431–7.

Article   CAS   PubMed   Google Scholar  

Royal College of Pathologists of Australasia. Massively Parallel Sequencing Implementation Guidelines. 2015. http://pathwiki.rcpaqap.com.au/pathwiki/index.php/Introduction . Last accessed September 15, 2022.

Miller DT, Lee K, Abul-Husn NS, Amendola LM, Brothers K, Chung WK, et al. ACMG SF v3.2 list for reporting of secondary findings in clinical exome and genome sequencing: a policy statement of the American College of Medical Genetics and Genomics (ACMG). Genet Med. 2023;25:100866.

Botkin JR, Belmont JW, Berg JS, Berkman BE, Bombard Y, Holm IA, et al. Points to consider: ethical, legal, and psychosocial implications of genetic testing in children and adolescents. Am J Hum Genet. 2015;97:6–21.

Green RC, Berg JS, Grody WW, Kalia SS, Korf BR, Martin CL, et al. ACMG recommendations for reporting of incidental findings in clinical exome and genome sequencing. Genet Med. 2013;15:565–74.

ACMG Board of Directors. ACMG policy statement: updated recommendations regarding analysis and reporting of secondary findings in clinical genome-scale sequencing. Genet Med. 2015;17:68–9.

Article   Google Scholar  

Kalia SS, Adelman K, Bale SJ, Chung WK, Eng C, Evans JP, et al. Recommendations for reporting of secondary findings in clinical exome and genome sequencing, 2016 update (ACMG SF v2.0): a policy statement of the American College of Medical Genetics and Genomics. Genet Med. 2017;19:249–55.

Strande NT, Riggs ER, Buchanan AH, Ceyhan-Birsoy O, DiStefano M, Dwight SS. et al. Evaluating the clinical validity of gene–disease associations: an evidence-based framework developed by the Clinical Genome Resource. Am J Hum Genet. 2017;100:895–906.

Rehm HL, Berg JS, Brooks LD, Bustamante CD, Evans JP, Landrum MJ, et al. ClinGen-the Clinical Genome Resource. N Engl J Med. 2015;372:2235–42.

DiStefano MT, Goehringer S, Babb L, Alkuraya FS, Amberger J, Amin M, et al. The Gene Curation Coalition: a global effort to harmonize gene–disease evidence resources. Genet Med. 2022;24:1732–42.

Clause AR, Taylor JP, Rajkumar R, Bluske K, Bennett M, Amendola LM, et al. Reactive gene curation to support interpretation and reporting of a clinical genome test for rare disease: experience from over 1,000 cases. Cell Genom. 2023;3:100258.

Richards S, Aziz N, Bale S, Bick D, Das S, Gastier-Foster J, et al. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med. 2015;17:405–24.

Brandt T, Sack LM, Arjona D, Tan D, Mei H, Cui H, et al. Adapting ACMG/AMP sequence variant classification guidelines for single-gene copy number variants. Genet Med. 2020;22:336–44.

Riggs ER, Andersen EF, Cherry AM, Kantarci S, Kearney H, Patel A, et al. Technical standards for the interpretation and reporting of constitutional copy-number variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics (ACMG) and the Clinical Genome Resource (ClinGen). Genet Med. 2020;22:245–57.

McCormick EM, Lott MT, Dulik MC, Shen L, Attimonelli M, Vitale O, et al. Specifications of the ACMG/AMP standards and guidelines for mitochondrial DNA variant interpretation. Hum Mutat. 2020;41:2028–57.

Berg JS, Foreman AKM, O’Daniel JM, Booker JK, Boshe L, Carey T, et al. A semiquantitative metric for evaluating clinical actionability of incidental or secondary findings from genome-scale sequencing. Genet Med. 2016;18:467–75.

Cochrane Reviews. Cochrane Library. https://www.cochranelibrary.com/ . Last accessed 2023.

NCCN. National Comprehensive Cancer Network - Home. https://www.nccn.org . Last accessed 2023.

PubMed. https://pubmed.ncbi.nlm.nih.gov/ .

Google. https://www.google.com/ .

Google Scholar. https://scholar.google.com/ .

Adam MP, Feldman J, Mirzaa GM, Pagon RA, Wallace SE, Bean LJ, et al., editors. GeneReviews®. Seattle, WA: University of Washington; 1993. http://www.ncbi.nlm.nih.gov/books/NBK1116/ .

Miller DT, Lee K, Chung WK, Gordon AS, Herman GE, Klein TE, et al. ACMG SF v3.0 list for reporting of secondary findings in clinical exome and genome sequencing: a policy statement of the American College of Medical Genetics and Genomics (ACMG). Genet Med. 2021;23:1381–90.

Miller DT, Lee K, Abul-Husn NS, Amendola LM, Brothers K, Chung WK, et al. ACMG SF v3.1 list for reporting of secondary findings in clinical exome and genome sequencing: a policy statement of the American College of Medical Genetics and Genomics (ACMG). Genet Med. 2022;24:1407–14.

Elfatih A, Mohammed I, Abdelrahman D, Mifsud B. Frequency and management of medically actionable incidental findings from genome and exome sequencing data: a systematic review. Physiol Genom. 2021;53:373–84.

Article   CAS   Google Scholar  

Nambot S, Sawka C, Bertolone G, Cosset E, Goussot V, Derangère V, et al. Incidental findings in a series of 2500 gene panel tests for a genetic predisposition to cancer: results and impact on patients. Eur J Med Genet. 2021;64:104196.

Cheung F, Birch P, Friedman JM, Elliott AM, Adam S, CAUSES Study, et al. The long-term impact of receiving incidental findings on parents undergoing genome-wide sequencing. J Genet Couns. 2022;31. https://pubmed.ncbi.nlm.nih.gov/35128755/ .

Hart MR, Biesecker BB, Blout CL, Christensen KD, Amendola LM, Bergstrom KL, et al. Secondary findings from clinical genomic sequencing: prevalence, patient perspectives, family history assessment, and health-care costs from a multisite study. Genet Med. 2019;21. https://pubmed.ncbi.nlm.nih.gov/30287922/ .

Landrum MJ, Lee JM, Benson M, Brown GR, Chao C, Chitipiralla S, et al. ClinVar: improving access to variant interpretations and supporting evidence. Nucleic Acids Res. 2018;46:D1062–7.

Cappellini MD, Fiorelli G. Glucose-6-phosphate dehydrogenase deficiency. Lancet. 2008;371:64–74.

Download references

Acknowledgements

We thank the ILS interpretation and reporting team: Krista Bluske, Maren Bennett, CMB, Matthew P. Brown, Amanda Buchanan, Brendan Burns, Nicole J. Burns, AC, Aditi Chawla, Amanda R. Clause, AJC, Katie Golden-Grant, Alka Malhotra, Becky Milewski, Samin A. Sajan, Zinayida Schlachetzki, Sarah Schmidt, Revathi Rajkumar, Julie P. Taylor, and Brittany Thomas. We thank Brittany Thomas for thoughtful review.

No specific funding.

Author information

These authors contributed equally: Laura M. Amendola, Anjana Chandrasekhar.

Authors and Affiliations

Medical Genomics Research, Illumina, Inc., San Diego, CA, 92122, USA

Carolyn M. Brown, Laura M. Amendola, Anjana Chandrasekhar, Akanchha Kesari, Erin Thorpe, Denise L. Perry, Ryan J. Taft & Alison J. Coffey

Oncology, Natera Inc., Austin, TX, 78753, USA

R. Tanner Hagelstrom

Scripps MD Anderson Cancer Center, San Diego, CA, 92121, USA

Gillian Halter

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization: LMA, CMB, AC, AJC, AK, RTH, ET. Data curation: LMA, CMB, AC, AJC. Formal Analysis: LMA, CMB, AC, AJC. Investigation: LMA, CMB, AC, AJC, GH, ISL interpretation and reporting team. Supervision: AJC, RTH, AK, DLP, RJT. Visualization: CMB, AC. Writing – original draft: LMA, CMB, AC, AJC. Writing – review & editing: LMA, CMB, AC, AJC, DLP, RJT, ET.

Corresponding authors

Correspondence to Carolyn M. Brown or Alison J. Coffey .

Ethics declarations

Competing interests.

All authors are shareholders of Illumina Inc. All authors are employees of Illumina Inc., except for GH and RTH who are former employees of Illumina Inc.

Ethical approval

Data analysis was completed under an institutional review board research exemption (ID Number: ICSL-001). Informed consent was not required since no human subjects were enrolled; all study samples were de-identified residual specimens left over from routine clinical care.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Title: ihope collaborator incidental findings survey, 41431_2024_1575_moesm2_esm.pdf.

Supplementary figure 1 title: Select responses from the “iHope collaborator incidental findings survey”, which examined physician and genetic counselor attitudes surrounding IFs in clinical genomic tes

41431_2024_1575_MOESM3_ESM.docx

Rights and permissions.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Brown, C.M., Amendola, L.M., Chandrasekhar, A. et al. A framework for the evaluation and reporting of incidental findings in clinical genomic testing. Eur J Hum Genet (2024). https://doi.org/10.1038/s41431-024-01575-1

Download citation

Received : 16 October 2023

Revised : 29 December 2023

Accepted : 20 February 2024

Published : 02 April 2024

DOI : https://doi.org/10.1038/s41431-024-01575-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Introduction
• Conclusions
• Article Information

Yields were calculated using 12 or more months as the time interval between the surveillance colonoscopy and the index colonoscopy. AA indicates advanced adenoma; NAA, nonadvanced adenoma.

Data Sharing Statement

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Lee JK , Roy A , Jensen CD, et al. Surveillance Colonoscopy Findings in Older Adults With a History of Colorectal Adenomas. JAMA Netw Open. 2024;7(4):e244611. doi:10.1001/jamanetworkopen.2024.4611

Manage citations:

© 2024

• Permissions

Surveillance Colonoscopy Findings in Older Adults With a History of Colorectal Adenomas

• 1 Division of Research, Kaiser Permanente Northern California, Oakland
• 2 Kaiser Permanente San Leandro Medical Center, San Leandro, California
• 3 Kaiser Permanente Washington Health Research Institute, Seattle
• 4 Kaiser Permanente Bernard J. Tyson School of Medicine, Pasadena, California
• 5 Rutgers Biological Health Sciences, Rutgers University, New Brunswick, New Jersey
• 6 Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas
• 7 Simmons Comprehensive Cancer Center, University of Texas Southwestern Medical Center, Dallas
• 8 Peter O’Donnell Jr School of Public Health, University of Texas Southwestern Medical Center, Dallas
• 9 Kaiser Permanente Southern California Department of Research and Evaluation, Pasadena
• 10 Department of Quality and Systems of Care, Kaiser Permanente Southern California, Pasadena
• 11 Kaiser Permanente Colorado Institute for Health Research, Aurora

Question   What are the colorectal cancer (CRC) and advanced neoplasia yields at surveillance colonoscopy among older patients with a history of colorectal adenoma, and do yields increase with age?

Findings   In this cross-sectional study of 9740 surveillance colonoscopies among 9601 adults aged 70 to 85 years with prior colorectal adenoma, CRC detection at surveillance was 0.3% overall and detection of advanced neoplasia was 12.0%. Yields were higher among patients with a prior advanced adenoma vs nonadvanced adenoma and did not increase significantly with age.

Meaning   In this study, CRC detection at surveillance colonoscopy was rare among older adults regardless of prior adenoma finding, whereas advanced neoplasia detection was more common and more likely in those with a prior advanced adenoma vs nonadvanced adenoma.

Importance   Postpolypectomy surveillance is a common colonoscopy indication in older adults; however, guidelines provide little direction on when to stop surveillance in this population.

Objective   To estimate surveillance colonoscopy yields in older adults.

Design, Setting, and Participants   This population-based cross-sectional study included individuals 70 to 85 years of age who received surveillance colonoscopy at a large, community-based US health care system between January 1, 2017, and December 31, 2019; had an adenoma detected 12 or more months previously; and had at least 1 year of health plan enrollment before surveillance. Individuals were excluded due to prior colorectal cancer (CRC), hereditary CRC syndrome, inflammatory bowel disease, or prior colectomy or if the surveillance colonoscopy had an inadequate bowel preparation or was incomplete. Data were analyzed from September 1, 2022, to February 22, 2024.

Exposures   Age (70-74, 75-79, or 80-85 years) at surveillance colonoscopy and prior adenoma finding (ie, advanced adenoma vs nonadvanced adenoma).

Main Outcomes and Measures   The main outcomes were yields of CRC, advanced adenoma, and advanced neoplasia overall (all ages) by age group and by both age group and prior adenoma finding. Multivariable logistic regression was used to identify factors associated with advanced neoplasia detection at surveillance.

Results   Of 9740 surveillance colonoscopies among 9601 patients, 5895 (60.5%) were in men, and 5738 (58.9%), 3225 (33.1%), and 777 (8.0%) were performed in those aged 70-74, 75-79, and 80-85 years, respectively. Overall, CRC yields were found in 28 procedures (0.3%), advanced adenoma in 1141 (11.7%), and advanced neoplasia in 1169 (12.0%); yields did not differ significantly across age groups. Overall, CRC yields were higher for colonoscopies among patients with a prior advanced adenoma vs nonadvanced adenoma (12 of 2305 [0.5%] vs 16 of 7435 [0.2%]; P  = .02), and the same was observed for advanced neoplasia (380 of 2305 [16.5%] vs 789 of 7435 [10.6%]; P  < .001). Factors associated with advanced neoplasia at surveillance were prior advanced adenoma (adjusted odds ratio [AOR], 1.65; 95% CI, 1.44-1.88), body mass index of 30 or greater vs less than 25 (AOR, 1.21; 95% CI, 1.03-1.44), and having ever smoked tobacco (AOR, 1.14; 95% CI, 1.01-1.30). Asian or Pacific Islander race was inversely associated with advanced neoplasia (AOR, 0.81; 95% CI, 0.67-0.99).

Conclusions and Relevance   In this cross-sectional study of surveillance colonoscopy yield in older adults, CRC detection was rare regardless of prior adenoma finding, whereas the advanced neoplasia yield was 12.0% overall. Yields were higher among those with a prior advanced adenoma than among those with prior nonadvanced adenoma and did not increase significantly with age. These findings can help inform whether to continue surveillance colonoscopy in older adults.

Colonoscopy is associated with reduced colorectal cancer (CRC) incidence and mortality through removal of adenomas, the main precursor lesions to CRC, and with decreased mortality through early detection and treatment of cancer. 1 - 15 Adenomas are found in nearly 40% of screening colonoscopies in the US, and after removal (polypectomy), guidelines recommend that patients undergo future surveillance colonoscopy. 14 - 18 However, guidelines provide little direction regarding the age at which colonoscopy surveillance is unlikely to be of substantial benefit and could be stopped. 18 , 19 Given the increasing aging population in the US and that nearly 5.6 million adults older than 75 years will undergo surveillance annually by 2024, 20 estimating the yield of surveillance colonoscopy is important for understanding the balance between potential benefits and known risks of colonoscopy with advancing age.

The risks of colonoscopy increase with age, particularly among those aged 75 years or older, and include heart attack, stroke, sedation-related adverse events (eg, aspiration pneumonia), bleeding, infection, and perforation. 21 In addition, the benefits of surveillance colonoscopy in older adults may be reduced because of a more limited life expectancy. 19 Also, in many settings, colonoscopy demand exceeds capacity, and therefore, it is important to direct procedures to those for whom potential benefits will likely outweigh possible harms. These arguments against surveillance colonoscopy in older adults must be weighed against findings that rates of CRC increase with age, at least among unscreened individuals. 22

In weighing the pros and cons of surveillance colonoscopy in older adults, information needed for shared decision-making between patients and clinicians includes the yields of CRC and advanced neoplasia at surveillance colonoscopy in this age group. Prior studies examining yields in older adults with a history of colorectal polyps have been limited by small sample sizes, limited racial and ethnic representation, and inability to examine yields stratified by prior colonoscopy findings and age. 23 - 29 To address this knowledge gap, we evaluated the surveillance colonoscopy yields of CRC and advanced neoplasia in patients 70 to 85 years of age with a prior adenoma finding from a large, demographically diverse, community-based US health care system. Yields were estimated overall (all ages combined), by age group (70-74, 75-79, and 80-85 years), and by the combination of age group and prior adenoma finding (advanced adenoma vs nonadvanced adenoma).

This cross-sectional study evaluated surveillance colonoscopy yields of CRC and advanced neoplasia in patients 70 to 85 years of age with a prior adenoma finding. This study adhered to the Strengthening the Reporting of Observational Studies in Epidemiology ( STROBE ) reporting guideline for cross-sectional studies. The study was approved by the Kaiser Permanente Northern California (KPNC) institutional review board with a waiver of informed consent because the research involved no more than minimal risk to participants and it could not practically be carried out without the requested waiver.

Study data were obtained from KPNC, and details of the population and screening practices have been described elsewhere. 30 The KPNC membership is demographically diverse and similar in socioeconomic characteristics to the region’s diverse census demographics, including the proportions of individuals with commercial insurance, Medicare, and Medicaid. 31 Thus, studies within this setting approximate community-based research within a demographically diverse population. 32

KPNC health plan members were eligible for the study if they were 70 to 85 years of age; underwent a surveillance colonoscopy between January 1, 2017, and December 31, 2019; and had a prior colonoscopy with an adenoma detected (hereafter, “index colonoscopy”) 12 or more months before their surveillance colonoscopy, colonoscopy and pathology reports available for each procedure, and at least 1 year of health plan enrollment prior to the surveillance procedure. Individuals were excluded if, prior to the surveillance colonoscopy, they had a diagnosis of CRC, hereditary CRC syndrome, and/or inflammatory bowel disease; had a prior colectomy; or their surveillance colonoscopy had an inadequate bowel preparation or was not complete to the cecum. The study sample included all patients who met the eligibility criteria.

The outcomes were CRC and advanced neoplasia (either CRC or advanced adenoma). In ascertaining outcome, the most advanced finding from the surveillance procedure was recorded (eg, for a patient diagnosed with both CRC and advanced adenoma, CRC was the recorded finding). Advanced adenoma diagnoses used pathology findings reported at or within 7 days after the procedure. To allow for additional diagnostic procedures for potentially inconclusive examinations, CRC diagnoses were ascertained at or within 180 days after the surveillance colonoscopy.

Data from clinical and administrative databases, including electronic health records, were used to obtain information on cohort member demographic characteristics, diagnoses, pathology findings, and procedures. Race and ethnicity were included in the analysis because some racial and ethnic groups in the US experience inequities in access to and utilization and quality of CRC screening and treatment as well as higher CRC incidence and mortality. 33 Race and ethnicity data were recorded as 1 of the following 8 categories as documented in the electronic health record: Hispanic; non-Hispanic Alaska Native or American Indian, Asian, Black, Pacific Islander, White, multiracial (reported multiple races), and unknown (race and ethnicity not reported).

Colonoscopies were identified using Current Procedural Terminology codes; International Classification of Diseases, Ninth Revision and International Statistical Classification of Diseases and Related Health Problems, Tenth Revision procedure codes; Healthcare Common Procedure Coding System codes; and site-specific codes. Colonoscopy indication (ie, screening, surveillance, diagnostic, and positive fecal immunochemical test result) was ascertained by a validated colonoscopy indication algorithm based on symptoms and conditions identified using electronic health records. 34 , 35 Colonoscopy quality measures (ie, extent of the examination and bowel preparation quality) were ascertained from colonoscopy reports using commercial natural language processing software (Linguamatics I2E; Linguamatics). This approach has been validated in comparison with manual record review. 36

Adenoma detection and histologic features were ascertained using Systematized Nomenclature of Medicine (SNOMED) coding in electronic pathology databases. Advanced adenoma was defined as a conventional adenoma with high-grade dysplasia or villous or tubulovillous histologic features or as any conventional adenoma 10 mm or greater in size; sessile serrated polyps, traditional serrated polyps, and hyperplastic polyps 10 mm or greater in size were not included in the definition. Nonadvanced adenoma was defined as any conventional adenoma less than 10 mm in size and without high-grade dysplasia or villous or tubulovillous histologic features. Advanced neoplasia was defined as any CRC or advanced adenoma. High-grade dysplasia has no specific SNOMED code and was identified using text string searches of pathology reports. Adenoma size of 10 mm or greater was obtained from a discrete data field within structured colonoscopy flow sheets. Colorectal cancer diagnoses were obtained from the KPNC cancer registry, which reports to the Surveillance, Epidemiology, and End Results program. Colorectal cancer was defined as an adenocarcinoma within the colon or rectum using Surveillance, Epidemiology, and End Results program codes 21040 and 21050; International Classification of Diseases for Oncology, Third Edition (ICD-O-3) site (topography) codes C18.0, C18.2-C18.9, C19.9, and C20.9; and ICD-O-3 histology (morphology) codes 8000, 8010, 8020, 8140, 8143, 8144, 8210, 8211, 8215, 8220, 8221, 8230, 8244, 8245, 8255, 8260-8263, 8323, 8480, 8481, 8490, 8510, 8560, and 8570-8574.

Summary statistics were used to describe the characteristics of patients who received a surveillance colonoscopy. Surveillance colonoscopy yields were calculated overall (for all ages), by age group (ie, 70-74, 75-79, and 80-85 years), and by both age group and prior adenoma finding (ie, advanced adenoma or nonadvanced adenoma). All surveillance procedures were considered in the yield calculations (ie, patients could contribute >1 procedure to the calculations). Differences in yield measures by age group were assessed using the χ 2 test of equal proportions. Trends in yields across age groups were evaluated using the Cochran-Armitage test for trend. In the primary analyses, yield measures were calculated using 12 months or longer as the time interval between the surveillance colonoscopy and the index colonoscopy. In sensitivity analyses, yield measures were calculated using 24 or more, 36 or more, 48 or more, and 60 or more months as the time interval to decrease the potential influence of higher-risk patients who may have been recommended to have a relatively early follow-up colonoscopy (ie, within the first few years).

Multivariable logistic regression was used to identify factors associated with advanced neoplasia detection at surveillance, and the odds ratio (OR) with 95% CI was used as an estimate of risk. The variables in the model were patient age (continuous, in years); sex (male or female); race and ethnicity, collapsed into 5 categories (Asian or Pacific Islander, Black, Hispanic, White, and remaining groups [Alaska Native or American Indian, multiracial, and unknown]); body mass index (BMI; calculated as weight in kilograms divided by height in meters squared) of less than 25, 25 to 29.9, or 30 or greater, ascertained at the measurement date closest to the date of the surveillance colonoscopy; tobacco smoking history (ever vs never or unknown); Charlson Comorbidity Index score (0, 1, or ≥2), ascertained in the calendar year before the surveillance colonoscopy; diabetes diagnosis any time prior to the surveillance colonoscopy (yes or no); family history of CRC (yes or no for any relative with CRC); and adenoma findings at the index colonoscopy (advanced or nonadvanced adenoma). In a post hoc analysis, we also included the time interval between the surveillance and index colonoscopies (continuous, in years). Two-sided P  < .05 indicated statistical significance, and analyses were conducted from September 1, 2022, to February 22, 2024, using SAS, version 9.3 (SAS Institute Inc).

Among 9601 patients 70 to 85 years of age who had an adenoma detected and a follow-up colonoscopy performed 12 or more months after the examination at which the adenoma was detected, 9740 surveillance colonoscopy procedures were performed from 2017 to 2019 ( Table 1 ); 5738 (58.9%) were performed in those aged 70 to 74 years, 3225 (33.1%) in those aged 75 to 79 years, and 777 (8.0%) in those aged 80 to 85. Among the total colonoscopies, 3845 (39.5%) were performed in females and 5895 (60.5%) in males. A total of 29 (0.3%) were in Alaska Native or American Indian patients, 1467 (15.1%) in Asian patients, 523 (5.4%) in Black patients, 899 (9.2%) in Hispanic patients, 28 (0.3%) in Pacific Islander patients, 6711 (68.9%) in White patients, 44 (0.5%) in multiracial patients, and 39 (0.4%) in patients with unknown race and ethnicity. The most prevalent BMI range category was 25 to 29.9 (3951 procedures [40.6%]). Nearly half of procedures were performed among patients who had never smoked tobacco (4864 [49.9%]), 4235 (43.5%) among patients with a Charlson Comorbidity Index score of 2 or higher, 2569 (26.4%) among patients who had diabetes, and 1590 (16.3%) among patients with a documented family history of CRC. The median time interval between the index and surveillance colonoscopies was 4.9 years (IQR, 3.2-5.4 years), and for 2305 (23.7%) of the surveillance colonoscopies, an advanced adenoma had been detected in the index procedure. These 9601 patients comprised the analytic cohort. Baseline characteristics by age group are shown in Table 1 .

Among the 9740 surveillance colonoscopies, CRC was the most advanced finding for 28 (0.3%), advanced adenoma for 1141 (11.7%), and advanced neoplasia (CRC or advanced adenoma) for 1169 (12.0%) ( Figure and Table 2 ). Surveillance colonoscopy yields did not increase significantly with age ( Figure and Table 2 ). Among those aged 70 to 74, 75 to 79, and 80 to 85 years, CRC yields were 0.2% (12 of 5738), 0.4% (13 of 3225), and 0.4% (3 of 777), respectively (test for trend, P  = .12). Advanced adenoma yields were 11.8% (679 of 5738), 11.3% (364 of 3225), and 12.6% (98 of 777), respectively (test for trend, P  = .99). Advanced neoplasia yields were 12.0% (691 of 5738), 11.7% (377 of 3225), and 13.0% (101 of 777), respectively (test for trend, P  = .79). In sensitivity analyses, yield estimates did not differ substantially when the time interval between the index and surveillance colonoscopies was restricted to 24 or more, 36 or more, 48 or more, and 60 or more months with the exception that when restricted to 60 or more months, the increase in CRC yield with age was significant ( Table 2 ).

At an interval of 12 or more months between colonoscopies, patients with a prior advanced adenoma had significantly higher yields of advanced neoplasia (380 of 2305 [16.5%] vs 789 of 7435 [10.6%]; P  < .001) and CRC (12 of 2305 [0.5%] vs 16 of 7435 [0.2%]; P  = .02) compared with those with a prior nonadvanced adenoma ( Figure and Table 3 ). The advanced neoplasia yields at the surveillance colonoscopy did not differ with increasing age for those with prior advanced adenoma (70-74 years: 219 of 1349 [16.2%]; 75-79 years: 124 of 759 [16.3%]; 80-85 years: 37 of 197 [18.8%]; test for trend, P  = .49) or nonadvanced adenoma (70-74 years: 472 of 4389 [10.8%]; 75-79 years: 253 of 2466 [10.3%]; 80-85 years: 64 of 580 [11.0%]; test for trend, P  = .82) ( Figure and Table 3 ). Similarly, CRC yields did not differ with increasing age for those with a prior advanced adenoma (70-74 years: 6 of 1349 [0.4%]; 75-79 years: 5 of 759 [0.7%]; 80-85 years: 1 of 197 [0.5%]; test for trend P  = .66) or nonadvanced adenoma (70-74 years: 6 of 4389 [0.1%]; 75-79 years: 8 of 2466 [0.3%]; 80-85 years: 2 of 580 [0.3%]; test for trend, P  = .10) ( Table 3 ). In sensitivity analyses, in surveillance colonoscopies both among patients with a prior advanced adenoma and patients with a prior nonadvanced adenoma, yield estimates did not differ substantially when the time interval between the surveillance colonoscopy and the index colonoscopy was restricted to 24 or more, 36 or more, 48 or more, and 60 or more months ( Table 3 ).

In a multivariable analysis ( Table 4 ), factors associated with the detection of advanced neoplasia at surveillance colonoscopy were prior advanced adenoma (adjusted OR [AOR], 1.65; 95% CI, 1.44-1.88), BMI of 30 or greater vs less than 25 (AOR, 1.21; 95% CI, 1.03-1.44), and having ever smoked tobacco (AOR, 1.14; 95% CI, 1.01-1.30). Compared with White patients, Asian or Pacific Islander patients were less likely to have advanced neoplasia detected at surveillance colonoscopy (AOR, 0.81; 95% CI, 0.67-0.99). In a post hoc analysis, the time interval between colonoscopies was not associated with advanced neoplasia detection and did not affect risk estimates for other factors.

In a large, integrated health care system, among 9740 surveillance colonoscopies in patients 70 to 85 years of age with a history of colorectal adenoma, detection of CRC or advanced neoplasia did not increase significantly with age. Overall, CRC detection was rare (0.3%), while detection of advanced neoplasia was more common (12.0%). Patients with a history of advanced adenoma vs nonadvanced adenoma were more likely to have CRC detected, though still rarely (0.5% vs 0.2%), and were more likely to have advanced neoplasia detected (16.5% vs 10.6%). These findings provide some of the first large-scale, community-based information on the yield of surveillance colonoscopy among older adults.

Prior studies of surveillance colonoscopy yield in older populations have reported conflicting results. Four studies reported that the prevalence of CRC and adenoma detection increased with age. 23 - 26 For example, a study using the New Hampshire Colonoscopy Registry showed that CRC yield at surveillance colonoscopy was low but increased from 0.4% to 0.6% and 0.8% among older adults aged 70-74, 75-79, and 80-84 years, respectively. 23 In contrast, 3 other studies reported that CRC prevalence decreased with age. 27 - 29 The inconsistency in prior studies may stem from the wide range of age groups studied (ie, 50-100 years of age), date of publication (given the improvement in colonoscopy techniques and adenoma detection in the past 10-15 years), different cohort sizes (ie, 80-42 611), and varying colonoscopy indications (ie, diagnostic, screening, and/or surveillance) in the study samples. We included only patients 70 to 85 years of age undergoing a surveillance colonoscopy following a colonoscopy in which adenomatous polyps were removed, and the yield estimates represent a contemporary population with regard to colonoscopy quality. Our surveillance colonoscopy yield estimates among older adults with a history of polyps are similar to or slightly higher than rates reported in the literature among individuals younger than 70 years, as would be expected given the older population in our study. 37 Based on a recent systematic review and meta-analysis, the yield of CRC among patients aged 50 to 70 years undergoing surveillance for a history of polyps ranged from 0.5% to 2.3%, with a pooled prevalence or yield of 1.4%. 37 In addition, yield of advanced polyps among patients aged 50 to 70 years undergoing surveillance for a history of polyps ranged from 2.9% to 24.4%, with a pooled prevalence or yield of 8.2%. 37

Current US guidelines do not provide a recommendation for the age to stop surveillance but advocate for the decision to be individualized based on benefits, risks, patient health status, and patient preferences. 17 , 18 The current study provides 2 key findings that can inform shared decision-making between patients and their clinicians. First, CRC detection at surveillance colonoscopy was rare among older adult patients with prior advanced or nonadvanced adenomas. Thus, for many older adults, particularly those with a prior nonadvanced adenoma, the low rate of CRC detection at surveillance may not justify the potential harms and burdens of colonoscopy that may increase with age. However, for some older adults with a predicted life expectancy of 10 or more years and without significant competing comorbidities, especially for those with a prior advanced adenoma, detection of early-stage CRC or advanced adenomas at surveillance could lead to earlier treatment and improved outcomes. Second, advanced adenoma detection at surveillance colonoscopy, which also did not increase with age, was more common than CRC detection; however, advanced adenomas themselves are not harmful to patients, and for the rare lesions that do progress to invasive cancer, the process takes several years. 38 Thus, among older adults with limited life expectancies due to comorbidities, few would likely benefit from the detection and removal of these polyps. The current data can help to estimate potential yields and benefits that can be considered vs the risks of sedation (eg, aspiration) and other potential colonoscopy-related harms (eg, perforation, major gastrointestinal bleeding), which increase with age. 21 , 39 - 41

Strengths of the study include the use of data from a large, demographically diverse, community-based health care system, which allowed access to comprehensive information about colonoscopy indications and findings. In addition, the large study size allowed for a specific focus on patients aged 70 years or older and stratifications by prior adenoma findings to provide patients and clinicians with granular data to inform individual decision-making regarding which older patients may be most likely to benefit from continued colonoscopy surveillance and which could potentially stop surveillance.

The study also had several limitations that should be considered. First, the study population was from a large, integrated health care system and limited to patients with a prior adenoma who had a complete colonoscopy with adequate bowel preparation; thus, the results may not be generalizable to unscreened populations or those with incomplete screening (who might be expected to be at higher risk). Second, the colonoscopy indication algorithm used may have misclassified some procedures as surveillance rather than diagnostic; however, validation studies have shown that it has high specificity (ie, 95%-96%) for detecting surveillance colonoscopies. 34 , 35 Third, our advanced adenoma definition did not include traditional serrated polyps or sessile serrated polyps given these do not have formal SNOMED codes; however, contemporary patients with such polyps have natural histories similar to those of patients with similarly sized polyps classified as traditional adenomas, particularly for small, serrated polyps. Fourth, family history data were collected through different data sources and aggregated in this analysis as any family history of CRC regardless of the degree of the relative, which may dilute its effect as a risk factor for CRC given that patients with a family history may have a greater risk of CRC. Fifth, caution should be used in drawing conclusions from our findings since the study was cross-sectional, the follow-up time was limited, and CRC development may take many years.

In this cross-sectional study, overall, CRC detection was rare and the yield of advanced neoplasia at surveillance colonoscopy was 12.0% among older adults in a large, community-based setting. Yields were higher in those with a prior advanced adenoma vs nonadvanced adenoma, and yields did not increase significantly with age. With current guidelines offering no specific age at which to stop surveillance colonoscopy, the study findings can inform clinicians and older patients regarding the potential benefits (or lack of benefits) of continuing with postpolypectomy surveillance in the context of the life expectancy of the patient and weighed against the rare but known harms of colonoscopy, which increase with advancing age and comorbidities.

Accepted for Publication: February 5, 2024.

Published: April 2, 2024. doi:10.1001/jamanetworkopen.2024.4611

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2024 Lee JK et al. JAMA Network Open .

Corresponding Author: Jeffrey K. Lee, MD, MPH, Division of Research, Kaiser Permanente Northern California, 2000 Broadway, Oakland, CA 94612 ( [email protected] ).

Author Contributions: Dr Lee had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs Lee and Roy share co–first authorship.

Concept and design: Lee, Roy, Jensen, Chan, Levin, Chubak, Corley.

Acquisition, analysis, or interpretation of data: Lee, Roy, Jensen, Zhao, Levin, Chubak, Halm, Skinner, Schottinger, Ghai, Burnett-Hartman, Kamineni, Udaltsova, Corley.

Drafting of the manuscript: Lee, Roy.

Critical review of the manuscript for important intellectual content: All authors.

Statistical analysis: Lee, Zhao, Udaltsova.

Obtained funding: Roy, Chubak, Halm, Skinner, Schottinger, Kamineni, Corley.

Administrative, technical, or material support: Schottinger, Ghai, Kamineni.

Supervision: Lee, Roy, Jensen, Chan, Levin.

Conflict of Interest Disclosures: Dr Levin reported receiving grants from the National Cancer Institute (NCI), National Institutes of Health (NIH) during the conduct of the study and receiving research support from Freenome, Inc. Dr Chubak reported receiving grants from the NCI during the conduct of the study and receiving grants from the NIH outside the submitted work. Dr Schottinger reported receiving grants from the NIH to the institution outside the submitted work. Dr Burnett-Hartman reported receiving grants from the NCI, NIH to the institution during the conduct of the study. Dr Kamineni reported receiving grants from the NCI during the conduct of the study. Dr Corley reported receiving grants from the NCI during the conduct of the study. No other disclosures were reported.

Funding/Support: The study was conducted within the Population-Based Research to Optimize the Screening Process consortium, funded by grant UM1 CA222035 from the NCI (Drs Chubak, Halm, Skinner, Schottinger, Kamineni, and Corley). The study was also supported by career development grant K07 CA212057 from the NCI (Dr Lee).

Role of the Funder/Sponsor: The NCI had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Data Sharing Statement: See the Supplement .

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

Comment on: “Aspirin for the Primary Prevention of Cardiovascular Diseases in Patients with Chronic Kidney Disease: An Updated Meta-analysis”

• Letter to the Editor
• Published: 02 April 2024

Cite this article

• Hussain Sohail Rangwala   ORCID: orcid.org/0009-0007-2167-3481 1 ,
• Hareer Fatima 1 &
• Burhanuddin Sohail Rangwala 1  

The Original Article was published on 01 February 2024

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Bellos I, Marinaki S, Lagiou P, et al. Aspirin for the primary prevention of cardiovascular diseases in patients with chronic kidney disease: an updated meta-analysis. Am J Cardiovasc Drugs. 2024. https://doi.org/10.1007/s40256-024-00630-y .

Article   PubMed   PubMed Central   Google Scholar  

Mann JFE, Joseph P, Gao P, et al. Effects of aspirin on cardiovascular outcomes in patients with chronic kidney disease. Kidney Int. 2023;103(2):403–10. https://doi.org/10.1016/j.kint.2022.09.023 .

Article   CAS   PubMed   Google Scholar  

Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94. https://doi.org/10.1016/j.jclinepi.2010.04.026 .

Article   PubMed   Google Scholar  

Pallikadavath S, Ashton L, Brunskill NJ, Burton JO, Gray LJ, Major RW. Aspirin for the primary prevention of cardiovascular disease in individuals with chronic kidney disease: a systematic review and meta-analysis. Eur J Prev Cardiol. 2022;28(17):1953–60. https://doi.org/10.1093/eurjpc/zwab132 .

Download references

Author information

Authors and affiliations.

Department of Medicine, Jinnah Sindh Medical University, Iqbal Shaheed Rd, Karachi, Pakistan

Hussain Sohail Rangwala, Hareer Fatima & Burhanuddin Sohail Rangwala

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Hussain Sohail Rangwala .

Ethics declarations

No external funding was used in the preparation of this letter.

Conflict of Interest

Hussain Sohail Rangwala, Hareer Fatima, and Burhanuddin Sohail Rangwala declare that they have no potential conflicts of interest that might be relevant to the contents of this letter.

Authors’ Contributions

The conceptualization was done by H.F. and H.S.R. The literature and drafting of the letter were conducted by H.F., H.S.R., and B.S.R. The editing and supervision were performed by H.S.R. All authors have read and agreed to the final version of the letter.

Data Availability Statement

Data sharing not applicable to this article as no datasets were generated in writing this letter.

Ethics approval

Not applicable.

Consent to Participate

Consent for publication, code availability, rights and permissions.

Reprints and permissions

About this article

Rangwala, H.S., Fatima, H. & Rangwala, B.S. Comment on: “Aspirin for the Primary Prevention of Cardiovascular Diseases in Patients with Chronic Kidney Disease: An Updated Meta-analysis”. Am J Cardiovasc Drugs (2024). https://doi.org/10.1007/s40256-024-00644-6

Download citation

Accepted : 18 March 2024

Published : 02 April 2024

DOI : https://doi.org/10.1007/s40256-024-00644-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Find a journal
• Publish with us
• Track your research

• Join the AMA
• Find learning by topic
• Free learning resources for members
• Certification
• Training for teams
• Why learn with the AMA?
• Marketing News
• Academic Journals
• Guides & eBooks
• Marketing Job Board
• Academic Job Board
• AMA Foundation
• Diversity, Equity and Inclusion
• Collegiate Resources
• Awards and Scholarships
• Sponsorship Opportunities
• Strategic Partnerships

We noticed that you are using Internet Explorer 11 or older that is not support any longer. Please consider using an alternative such as Microsoft Edge, Chrome, or Firefox.

How Fast-Fashion Copycats Hurt—and Help—High-End Fashion Brands

Soo Yon Ryu and Ceylin Petek Ertekin

Journal of Marketing Research Scholarly Insights are produced in partnership with the AMA Doctoral Students SIG – a shared interest network for Marketing PhD students across the world.

The global fashion market has achieved unprecedented growth, being valued at 1.74 trillion USD as of 2023, and it’s expected to expand further to 1.94 trillion USD by 2027 (Statista 2023). In the midst of reaching new heights, two colossal forces of the industry—high-end luxury labels and fast-fashion giants—remain in conflict. As soon as the former spurs a novel trend, the latter produces a copycat almost immediately. Despite the longstanding and growing tension between the two fashion titans, a critical question remains: do these fast-fashion copycats truly threaten the reign of high-end brands, or could they unexpectedly fuel their popularity?

In a recent Journal of Marketing Research study , researchers Zijun (June) Shi, Xiao Liu, Dokyun Lee, and Kannan Srinivasan discuss the dynamics between traditional high-end fashion brands and fast-fashion brands like Zara and H&M, with a focus on the impact of copycats on the high-end market. Despite the growing interest in and concerns about copycats, there has been limited empirical investigation into the subject due to challenges in quantifying fashion and obtaining consumer-level fashion choice data.

The authors overcome these challenges by incorporating deep learning techniques to measure the compatibility and distinctiveness of fashion items and analyzing image-based user-generated content from a fashion-focused online community. Driven from a rich dataset comprising 10,262 users and 64,681 fashion posts over four years, the authors find that fast-fashion copycats may negatively and positively affect high-end brands. Although copycats may harm high-end brands to some extent (a cannibalization effect), they can also be helpful for growth (a market expansion effect). Specifically, the market expansion effect is driven via both static and dynamic mechanisms: mix-and-match styling (a static mechanism), value enhancement via increased popularity on social media, and cost reduction via sponsorship opportunities for popular users (dynamic mechanisms).

Although copycats may harm high-end brands to some extent (a cannibalization effect), they can also be helpful for growth (a market expansion effect).

Furthermore, using a dynamic structural model incorporating an additional data set of 1,380 original–copycat pair images, the authors show that the impact of prohibiting copycats depends on the threshold of similarity. If only extremely similar copycats are banned, high-end brands benefit, as cannibalization seemingly dominates the market expansion effect among remarkably similar copycats. However, if moderately similar items are also banned, high-end brands suffer, as the market expansion effect seemingly dominates the cannibalization effect among moderately similar fast-fashion items. These findings provide insights for managers and policymakers in addressing the magnitude and limitations of copycats’ impact on high-end brands.

We interviewed the authors to discuss further actionable implications, generalizability to other categories, challenges in methodology, and much more:

Q: Based on the trickle-down theory of trend adoption (Simmel 1904), a central role of fast-fashion brands may be the affordable replication of higher-end brands. Considering the fundamental role of imitation in fast fashion, how do you anticipate copyright policies being enforced in this context? Has there been policy implementation addressing the pertinent issue? If public policy is falling short in addressing the imitation problem, what approaches should managers of luxury brands employ to confront the obstacles and potential advantages posed by fast-fashion copycats?

A: Currently the copyright policy does not protect fashion designs. From the managers’ perspective, our research provides helpful implications for luxury brands. That is, copycats would benefit the luxury items if the similarity is not very high. However, the cannibalization effect may dominate if the copycats are too similar to the original design. Therefore, instead of trying to eliminate all copycats, one approach the premium brands can take is to make it difficult for fast fashion to copy their product design to maintain a low similarity.

Q: Do you view your findings to be applicable to other retail categories like consumer electronics, where lower-end brands often replicate designs of higher end brands? If not, what is unique about fashion products that set it apart?

A: It requires additional empirical study to evaluate the effect in other retail categories. Fashion products are unique from several perspectives. First, visual features are key product characteristics, which we quantify and use to gauge the similarity between the original and copycat products in our study. Second, consumers use fashion products differently from other products in the sense that they need to mix and match multiple items to form a complete ensemble. This mix-and-match behavior is also a key mechanism that explains our findings of fast-fashion copycats’ impact on high-end brands.

Q: You used state-of-art machine learning techniques to quantify user-generated visual content. What were some challenges you faced and addressed in quantifying the subjective nature of fashion? Were there alternative ways you have considered that may have yielded different insights?

A: A big challenge was quantifying the compatibility and similarity between two product items. While similarity is relatively more straightforward to measure because we can use many existing tools from the computer vision literature, compatibility is much more challenging. We initially tried transferred learning by collecting human-labeled data and consumer co-purchase data to fine-tune the deep neural network, which unfortunately cannot fully avoid the issue of subject representativeness. The results could be biased if the measurement error is non-negligible. Later we decided to switch to applying a more cutting-edge model trained on similar tasks (i.e., rating compatibility of clothing items).

Q: This study observes data from 2013 to 2017 from a fashion-focused social media platform that has now become obsolete. Since then, the luxury market has grown significantly, and the dynamics of social media have become more diversified. If one were to conduct a similar study in the current market, what additional factors would you consider to reflect the changes? How would you conduct the study beyond social media users to overcome limited generalizability and self-selection bias?

A: The methods, models, and mechanisms we proposed in the paper are generalizable. To understand the copycat effects in more recent markets or beyond social media, one only needs to apply our method to more recently collected datasets or those collected beyond social media.

Q: Luxury brands are increasingly collaborating with fast fashion brands (e.g. H&M x Mugler, Target x Missoni). How do you see these collaborations fitting into your findings? Do you view this trend as a potential solution to the copycat issue or a risk to luxury brands?

A: This could be a valid solution. Alternatively, as discussed in the paper, high-end brands could enable consumers to mix and match items from different price tiers by launching affordable versions of their own high-end styles, capturing the unserved demand rather than leaving it to fast-fashion brands. That said, it is worth noting that both approaches may erode the high-end brand’s value. However, the net effect is unclear without additional information and further empirical study.

Simmel, Georg (1904), “Fashion,” International Quarterley , 10, 130-55.

Statista (2023), “Revenue of the apparel market worldwide from 2014 to 2027,” (accessed December 2023), https://www.statista.com/forecasts/821415/value-of-the-global-apparel-market .

Read the Full Study for Complete Details

Read the full article:

Zijun (June) Shi, Xiao Liu, Dokyun Lee, and Kannan Srinivasan, “ How Do Fast-Fashion Copycats Affect the Popularity of Premium Brands? Evidence from Social Media ,” Journal of Marketing Research , 60 (6), 1027–1051.

Go to the  Journal of Marketing Research

The Vicarious Haptic Effect: Your Hand is Mine and I Like What It’s Holding

Do You Feel Pretty? Consumers Make More Deliberate Choices When They Perceive Themselves as More Attractive

Self(ie) Disclosure: How Selfies Shape Brand Interactions

Soo Yon Ryu is a doctoral student in marketing, University of Florida, USA.

Ceylin Petek Ertekin is a doctoral student in marketing, London School of Economics and Political Science, UK.

By continuing to use this site, you accept the use of cookies, pixels and other technology that allows us to understand our users better and offer you tailored content. You can learn more about our privacy policy here

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Generalizability

Clinical and epidemiologic investigations are paying increasing attention to the critical constructs of “representativeness” of study samples and “generalizability” of study results. This is a laudable trend and yet, these key concepts are often misconstrued and conflated, masking the central issues of internal and external validity. The authors define these issues and demonstrate how they are related to one another and to generalizability. Providing examples, they identify threats to validity from different forms of bias and confounding. They also lay out relevant practical issues in study design, from sample selection to assessment of exposures, in both clinic-based and population-based settings.

Only to the extent we are able to explain empirical facts can we obtain the major objective of scientific research, namely not merely to record the phenomena of our experience, but to learn from them, by basing upon them theoretical generalizations which enable us to anticipate new occurrences and to control, at least to some extent, the changes in our environment. 1(p12)

“This study sample is not representative of the population!” “Our results are not generalizable …” Such comments are increasingly familiar but what exactly do they mean? How do study design, subject ascertainment, and “representativeness” of a sample affect “generalizability” of results? Do study results generalize only from statistically drawn samples of a common underlying population? Has “lack of generalizability” become the low-hanging fruit, ripe for plucking by the casual critic?

INTERNAL AND EXTERNAL VALIDITY

Confusion around generalizability has arisen from the conflation of 2 fundamental questions. First, are the results of the study true, or are they an artifact of the way the study was designed or conducted; i.e., is the study is internally valid? Second, are the study results likely to apply, generally or specifically, in other study settings or samples; i.e., are the study results externally valid?

Thoughtful study design, careful data collection, and appropriate statistical analysis are at the core of any study's internal validity. Whether or not those internally valid results will then broadly “generalize,” to other study settings, samples, or populations, is as much a matter of judgment as of statistical inference. The generalizability of a study's results depends on the researcher's ability to separate the “relevant” from the “irrelevant” facts of the study, and then carry forward a judgment about the relevant facts, 2 which would be easy if we always knew what might eventually turn out to be relevant. After all, we generalize results from animal studies to humans, if the common biologic process or disease mechanism is “relevant” and species is relatively “irrelevant.” We also draw broad inferences from randomized controlled trials, even though these studies often have specific inclusion and exclusion criteria, rather than being population probability samples. In other words, generalization is the “big picture” interpretation of a study's results once they are determined to be internally valid.

SAMPLING AND REPRESENTATIVENESS

The statistical concepts of sampling theory and hypothesis testing have become intermingled with the notion of generalizability. Strict estimation of quantities based on a probability sample of a “population,” vs assessing all members of that population, remained an object of considerable argument among statisticians until the early 20th century. 3 Sampling was adopted of necessity because studying the entire population was not feasible. Fair samples must provide valid estimates of the population characteristics being studied. This quite reasonable concept evolved in common usage so that “population” became synonymous with “all persons or all cases.” It followed that to achieve representative and generalizable sample estimates, a probability sample of “all” must be drawn. Logically, then, “all” must somehow be enumerated before representative samples can be drawn. The bite of the vicious circle becomes obvious when “all” literally means all in a country or continent. Yet enumeration may be achievable when care is taken to establish more finite population boundaries.

Statisticians Kruskal and Mosteller 3 – 6 conducted a detailed examination of nonscientific, “extrastatistical scientific,” and statistical literature to classify uses of the term representative sample or sampling. Those meanings are 1) “general, unjustified acclaim for the data”; 2) “absence (or presence) of selective forces”; 3) “mirror or miniature of the population”; 4) “typical or ideal case … that represents it (the population) on average”; 5) “coverage of the population … (sample) containing at least one item from each stratum …”; 6) “a vague term to be made precise” by specification of a particular statistical sampling scheme, e.g., simple random sampling. In statistical literature, representative sampling meanings include a) “a specific sampling method”; b) “permitting good estimation”; and c) “good enough for a particular purpose.” 4 The conflicts and ambiguities among the above uses are obvious, but how do we seek clarity in our research discourse?

POPULATIONS, CLINICS, AND BOUNDARIES

So is there in fact any value to population-based studies (Indeed there is!), and if so, how should we define a “population”? We first define it by establishing its boundaries (e.g., counties, insurance memberships, schools, voter registration lists). The population is made up entirely of members with disease (cases) and members without disease (noncases), leaving nobody out. Ideally, we would capture and study all cases, as they occur. As a comparison group, we would also include either all noncases, or a probability sample of noncases. 7 The choice of “boundaries” for a study population influences internal and external validity. If we deliberately or inadvertently “gerrymander” our boundaries, so that the factor of interest is more (or less) common among cases than among noncases, the study base will be biased and our results will be spurious or misleading.

Adequately designed population-based studies minimize the possibility that selection factors will have unintended adverse consequences on the study results. Further, since any effect we might measure depends as much on the comparison group as it does on the case group, appropriate selection is no less important for the noncases than it is for cases. This is true whether the study is clinic-based or population-based. Population-based research anchors the comparison group to the cases.

Clinic-based investigations are exemplified by those conducted at Alzheimer's Disease Research Centers (ADRCs). They typically examine high-risk, family-based, clinic-based, or hospital-based groups, to observe association with treatment or disease. This is an efficient means to facilitate in-depth study of “clean” diagnostic subgroups. The external validity of these studies rests on the judgment of whether the subject selection process itself could have spuriously influenced the results. This determination is often harder in clinic-based studies than in population-based studies. Replication in an independent sample is therefore key, but replication is more elusive and difficult with clinic-based studies, as we discuss later.

Regardless of whether the study sample is clinic-based or population-based, how well and completely we identify “disease” (including preclinical or asymptomatic disease), not only in our case group, but also among those in our comparison group, can adversely impact results. For example, consider a study of Alzheimer disease (AD) in which, unbeknownst to the subjects as well as the investigators, the cognitively normal control group includes a large proportion of persons with underlying AD pathology. The resulting diagnostic misclassification, caused by including true “cases” among the noncases, would spuriously distort and weaken the observed results. This distortion can happen in clinic-based or population-based studies; it is a matter of internal validity tied to diagnostic accuracy, rather than an issue of representativeness or generalizability.

Bias causes observed measurements or results to differ from their true values because of systematic, but unintended, “errors,” for example, in the way we ascertain and enroll study subjects (selection bias), or the way we collect data from them (information bias). Statistical significance of study results, regardless of p value, is completely irrelevant as a means of evaluating results when bias is active.

Selection bias.

Selection bias is often subtle, and requires careful thought to discern its potential effect on the hypotheses being tested. For example, would selection bias render clinic-based ADRC study results suspect, if not invalid? Unfortunately, the answer is not simple; it depends on what is being studied and whether “selection” into the ADRC study distorts the true association. There are numerous advantages to recruiting study participants from specialized memory disorder clinics, as in the typical ADRC. Both AD cases and healthy controls are selected (as volunteers or referrals) under very specific circumstances that ensure their contribution to AD research. They either have (cases) or do not have (controls) the clinical/pathologic features typical of AD. Cases fulfill the research diagnostic criteria for AD, they have “reliable informants” who will accompany them to clinic visits; neither cases nor controls can have various exclusionary features (e.g., comorbid stroke or major psychiatric disorder); all are motivated to come to the clinic and participate fully in the research, including neuroimaging and lumbar puncture; many are eager to enter clinical trials, and many consent to eventual autopsy. AD cases who fit the above profile are admirable for their enthusiasm and altruism, but may not be typical, nor a probability sample of all AD cases in the population base from whence they came. The differential distribution of study factors between AD cases who did and did not enroll could give us an indication of whether bias may be attenuating or exaggerating the specific study results, if we were able to obtain that information. Therefore, the astute reader asks : “Can the underlying population base, from which the subjects came, be described? Might the population base's established boundaries or inclusion characteristics have influenced the results? Was subject enrollment in any way influenced by the factors being studied?” In a clinic-based study it is seldom easy to describe the unenrolled cases (or unenrolled noncases) from the underlying population base in order to make such comparisons. It helps internal validity very little to claim that the enrollees' age, race, and sex distributions are in similar proportions to the population of the surrounding county, if age, race, and sex have little to do with the factor being studied, and if participation is differentially associated with the factors being studied.

Note that population-based studies are not inherently protected from bias; individuals sampled from the community, who are not seeking services, may consent or refuse to participate in research, and their willingness to participate is unlikely to be random. If we were concerned about selection bias in a study examining pesticide exposure as a risk factor for Parkinson disease (PD), we might ask, “Were PD cases who had not been exposed to pesticides more (or less) likely to refuse enrollment in our study than PD cases who had been exposed?”

Selection bias may be not just inadvertent but also unavoidable. Some years ago, a startling finding 8 was reported that AD cases who volunteered or were referred to an ADRC were significantly more likely to carry the APOE*4 genotype than were newly recognized AD cases captured through surveillance of a health maintenance organization population base within the same metropolitan area. The ADRC sample had yielded a biased overestimate of APOE*4 allele frequency, and of its estimated relative risk, because ADRC cases were inadvertently selected on the basis of age, and it was unnoticed that the likelihood of carrying an APOE*4 allele decreases with age. There is no way the ADRC investigators could have detected this inadvertent selection bias had they not also had access to a population sample from the same base. A later meta-analysis of APOE*4 allele effects quantified the relationship between age and risk of AD associated with APOE alleles, and showed that AD risk due to APOE*4 genotype is lower in population samples than in specialty clinic samples. 9 APOE allele frequency also could be influenced by study recruitment. Family history of AD seems to promote participation in both clinical and population-based studies involving memory loss, and is also associated with APOE*4 frequency, thereby potentially biasing the magnitude of APOE effect.

Survival bias is a form of selection bias that is beyond the control of the selector. For example, some African populations have high APOE*4 frequency but have not shown an elevated association between APOE*4 and AD. 10 , 11 While there could be multiple reasons for this paradox, one possibility is that individuals with the APOE*4 genotype had died of heart disease before growing old enough to develop dementia.

Prevalence bias (length bias) is similar to survival bias. In the 1990s, numerous case-control studies showed a protective effect of smoking on AD occurrence. 12 Assume that both AD and smoking shorten life expectancy and that AD cases enrolled in those studies some time after symptom onset. If age alone was the basis for potential selection bias, smoking should cause premature mortality equally among those who are and those who are not destined to develop AD. However, there is another aspect of selection bias called prevalence or length bias: at any given time, prevalent, i.e., existing, cases are those whose survival with disease (disease duration) was of greater length. If smokers with AD die sooner after AD onset than nonsmokers with AD, those prevalent AD cases available for study would “selectively” be nonsmokers. A scenario known as “competing risks” occurs when smoking influences the risk both of death and of AD. 13 This would enhance the observed excess of smoking among “controls” and thereby inflate the apparent protective association between smoking and AD. Subsequently, longitudinal studies of smokers and nonsmokers showed an increased risk of AD incidence associated with smoking, 12 suggesting that selection bias might have explained the earlier cross-sectional study results.

Information bias.

Information bias (data inaccuracy) can occur if we measure or determine the outcome or the exposure with substantial error or if the outcome or exposure is measured differently between comparison groups. Here, the reader must ask “Was information on the study factors and covariates gathered in a fair and equal manner for all subjects?” For example, suppose we obtain the history of previous head trauma, from spouses of the cases, but by self-report from the controls. The frequency of head trauma could be systematically different between groups because of whom we asked, rather than because of their true occurrence. Many earlier case-control studies showed an association between AD and previous history of head trauma. 14 This finding was not replicated in a subsequent study based on prospective data from a comprehensive population-based record-linkage system. 15 Here, data about head injury were recorded in the same way from all subjects before the onset of dementia; when both selection bias (including length bias) and information bias were eliminated, the association was no longer present. More recently the issue has raised its battered head once again, but such studies should also be mindful of the methodologic lessons of the past.

CONFOUNDING

Having done our best to avoid bias, how do we account for the simultaneous effects of other factors that could also cause the disease? Consider a study of diabetes as a risk factor for cognitive decline. Both diabetes and cognitive decline are associated with age, family history, and cerebrovascular disease. The effects of these other factors could distort our results, if they were unequally distributed between those with and without diabetes. This mixing of effects is called confounding. Similarly, in designing a study examining pesticide exposure as a risk factor for PD, we would be concerned about other risk or protective factors for PD which might themselves be associated with pesticide use. 16 A common additional exposure in rural farming areas is head trauma, 17 which arguably may increase risk of PD. 18 If head trauma was a causal risk factor and was distributed unequally between the pesticide-exposed and nonexposed groups, a spurious impression could be created about the risk associated with pesticide exposure.

If we proactively collected data on potential confounders, their effects could be “adjusted for” (equalized statistically between comparison groups) in the analysis, and can be similarly be “adjusted” in replication studies. Adjustment indicates ceteris paribus (holding all else constant): it statistically equalizes or removes the effect of the confounding factors (e.g., head trauma) so that the factor of interest (e.g., pesticide exposure) can be evaluated for its own effect. Note: bias (unlike confounding) can rarely be adjusted away.

REPLICATION

Replication of results in independent samples supports both the internal validity and the generalizability of the original finding, and is now required for publication of genetic association studies. If 2 similar studies' results do not agree, one does not necessarily refute the other; however, several similar studies failing to replicate the original would weigh heavily against the original result. We do not expect all risk factor studies to have identical results because risk factor frequencies may be differentially distributed among populations. Sample variability does not rule out generalizability, a priori, but the potential effects of bias and confounding must not be ignored.

GENERALIZABILITY AND POWER

Finally, another issue often wrongly subsumed under generalizability is related to the statistical power to observe an association if one truly exists. For example, a study of head trauma as a risk factor for dementia should be carried out in a sample where there is both sufficient head trauma and sufficient dementia for an association (if present) to be detected. A sample of young football players may have the former but not the latter 19 ; a sample of elderly nuns 20 may have the latter but not the former; a sample of retired football players may have both 21 ; a sample of aging military veterans may also have both, but there may be potential confounding factors associated with military service, such as other injuries, depression, or post-traumatic stress. 22 Thus, studies in different samples may not replicate one another's results with regard to head trauma and dementia not because the association changes but because of varying exposure or outcome frequency.

THE SMOKING GUN

We close with the one of the most influential articles of the 20th century, to demonstrate how even very narrowly defined study samples may provide widely generalizable results if conducted with an eye to rigorous internal validity. Entitled “The mortality of doctors in relation to their smoking habits: a preliminary report,” 23 this 1954 article by Doll and Hill concerned the association between lung cancer and cigarette smoking in British physicians. All 59,600 physicians in the Medical Register at the time were sent a questionnaire on their smoking habits. The investigators excluded physicians who did not return usable responses, and also women physicians, and physicians aged <35 years, because of their low expected frequency of lung cancer deaths. The remaining sample was a male, physician cohort of 24,389, about 40% of those in the Medical Register. During the 29-month follow-up, investigators observed only 36 confirmed lung cancer deaths, occurring at rates of 0.00 per 1,000 in nonsmokers, and 1.14 per 1,000 among smokers of 25 or more grams of tobacco per day. The lung cancer death rate was dose-dependent on amount smoked, but the same relationship with tobacco dose was observed neither for 5 other disease comparison groups, nor for all causes of death. Further, study cohort had an all-cause death rate of 14.0 per 1,000 per year as compared to 24.6 per 1,000 for men of all social classes and similar age. 23

Surely, that study provided a veritable feast of low-hanging fruit for critics focused on generalizability. With such a select study sample, would the results not be so specific and isolated that none would generalize to groups other than male British physicians? Undaunted, Doll and Hill focused on internal validity, considering whether their study-defined boundaries and method of subject selection could have created a spurious association between smoking and lung cancer death. They reasoned that the initially nonresponding physicians may have over-represented those already close to death, causing the observed death rate in the short term to be lower than the general population. More importantly, they asked whether such a difference in mortality within their sample could have caused the dose-response “gradient” between amount smoked and lung cancer death rate. “For such an effect we should have to suppose that the heavier smokers who already knew that they had cancer of the lung tended to reply more often than the nonsmokers or lighter smokers in a similar situation. That would not seem probable to us.” 23(p1454) This study has been replicated in many other population- and clinic-based studies. It has been generalized, in the broad scientific sense, to a variety of other groups, populations, and settings, despite the decidedly “nonrepresentative” nature of the study group and its specific boundaries. Doll and Hill focused on how, and how much, the definition of their study group and its characteristics could have influenced their results. That is, they considered how the effects of subject selection (i.e., selection bias), data accuracy (i.e., information bias), and unequal distribution of other risk/protective factors between comparison groups (i.e., confounding) could have threatened the study's internal validity. They also considered “power” when they excluded younger men and women. In this study, the “relevant” factor concerned the potential carcinogenic effect of tobacco smoke on human lung tissue.

Would the designs and findings of similar studies among restricted groups, nonrepresentative of the universe, be as readily accepted today? Would current readers question whether the results from British physicians would also apply to Wichita linemen or to real housewives from New Jersey? The British physicians were likely different in many ways from groups to which we might want to “generalize” the principal results. But they were not fundamentally different in ways that would affect our conclusions about the effect of tobacco smoke on lung tissue and ultimate mortality.

Science proceeds by replication and by generalization of individual study results into broader hypotheses, theories, or conclusions of fact. Establishing study boundaries and conducting “population-based” research within them enhances both internal validity and the likelihood that results may apply to similar and dissimilar groups. However, studies of specifically defined groups may also generalize to extend our knowledge. We could yield to temptation and seize the low-hanging fruit, vaguely challenging a study on grounds of generalizability. But then we would miss the forest for the trees.

AUTHOR CONTRIBUTIONS

Walter A. Kukull, PhD, made substantive contribution to the design and conceptualization and interpretations and was responsible for the initial draft and revising the manuscript. Mary Ganguli, MD, MPH, made substantive contribution to the design and conceptualization and interpretations and contributed to revising the manuscript.

The authors report no disclosures relevant to the manuscript. Go to Neurology.org for full disclosures.