types of research validity and reliability

• How it works

Reliability and Validity – Definitions, Types & Examples

Published by Alvin Nicolas at August 16th, 2021 , Revised On October 26, 2023

A researcher must test the collected data before making any conclusion. Every  research design  needs to be concerned with reliability and validity to measure the quality of the research.

What is Reliability?

Reliability refers to the consistency of the measurement. Reliability shows how trustworthy is the score of the test. If the collected data shows the same results after being tested using various methods and sample groups, the information is reliable. If your method has reliability, the results will be valid.

Example: If you weigh yourself on a weighing scale throughout the day, you’ll get the same results. These are considered reliable results obtained through repeated measures.

Example: If a teacher conducts the same math test of students and repeats it next week with the same questions. If she gets the same score, then the reliability of the test is high.

What is the Validity?

Validity refers to the accuracy of the measurement. Validity shows how a specific test is suitable for a particular situation. If the results are accurate according to the researcher’s situation, explanation, and prediction, then the research is valid. 

If the method of measuring is accurate, then it’ll produce accurate results. If a method is reliable, then it’s valid. In contrast, if a method is not reliable, it’s not valid. 

Example:  Your weighing scale shows different results each time you weigh yourself within a day even after handling it carefully, and weighing before and after meals. Your weighing machine might be malfunctioning. It means your method had low reliability. Hence you are getting inaccurate or inconsistent results that are not valid.

Example:  Suppose a questionnaire is distributed among a group of people to check the quality of a skincare product and repeated the same questionnaire with many groups. If you get the same response from various participants, it means the validity of the questionnaire and product is high as it has high reliability.

Most of the time, validity is difficult to measure even though the process of measurement is reliable. It isn’t easy to interpret the real situation.

Example:  If the weighing scale shows the same result, let’s say 70 kg each time, even if your actual weight is 55 kg, then it means the weighing scale is malfunctioning. However, it was showing consistent results, but it cannot be considered as reliable. It means the method has low reliability.

Internal Vs. External Validity

One of the key features of randomised designs is that they have significantly high internal and external validity.

Internal validity  is the ability to draw a causal link between your treatment and the dependent variable of interest. It means the observed changes should be due to the experiment conducted, and any external factor should not influence the  variables .

Example: age, level, height, and grade.

External validity  is the ability to identify and generalise your study outcomes to the population at large. The relationship between the study’s situation and the situations outside the study is considered external validity.

Also, read about Inductive vs Deductive reasoning in this article.

Looking for reliable dissertation support?

We hear you.

• Whether you want a full dissertation written or need help forming a dissertation proposal, we can help you with both.
• Get different dissertation services at ResearchProspect and score amazing grades!

Threats to Interval Validity

Threats of external validity, how to assess reliability and validity.

Reliability can be measured by comparing the consistency of the procedure and its results. There are various methods to measure validity and reliability. Reliability can be measured through  various statistical methods  depending on the types of validity, as explained below:

Types of Reliability

Types of validity.

As we discussed above, the reliability of the measurement alone cannot determine its validity. Validity is difficult to be measured even if the method is reliable. The following type of tests is conducted for measuring validity. 

Does your Research Methodology Have the Following?

• Great Research/Sources
• Perfect Language
• Accurate Sources

If not, we can help. Our panel of experts makes sure to keep the 3 pillars of Research Methodology strong.

How to Increase Reliability?

• Use an appropriate questionnaire to measure the competency level.
• Ensure a consistent environment for participants
• Make the participants familiar with the criteria of assessment.
• Train the participants appropriately.
• Analyse the research items regularly to avoid poor performance.

How to Increase Validity?

Ensuring Validity is also not an easy job. A proper functioning method to ensure validity is given below:

• The reactivity should be minimised at the first concern.
• The Hawthorne effect should be reduced.
• The respondents should be motivated.
• The intervals between the pre-test and post-test should not be lengthy.
• Dropout rates should be avoided.
• The inter-rater reliability should be ensured.
• Control and experimental groups should be matched with each other.

How to Implement Reliability and Validity in your Thesis?

According to the experts, it is helpful if to implement the concept of reliability and Validity. Especially, in the thesis and the dissertation, these concepts are adopted much. The method for implementation given below:

Frequently Asked Questions

What is reliability and validity in research.

Reliability in research refers to the consistency and stability of measurements or findings. Validity relates to the accuracy and truthfulness of results, measuring what the study intends to. Both are crucial for trustworthy and credible research outcomes.

What is validity?

Validity in research refers to the extent to which a study accurately measures what it intends to measure. It ensures that the results are truly representative of the phenomena under investigation. Without validity, research findings may be irrelevant, misleading, or incorrect, limiting their applicability and credibility.

What is reliability?

Reliability in research refers to the consistency and stability of measurements over time. If a study is reliable, repeating the experiment or test under the same conditions should produce similar results. Without reliability, findings become unpredictable and lack dependability, potentially undermining the study’s credibility and generalisability.

What is reliability in psychology?

In psychology, reliability refers to the consistency of a measurement tool or test. A reliable psychological assessment produces stable and consistent results across different times, situations, or raters. It ensures that an instrument’s scores are not due to random error, making the findings dependable and reproducible in similar conditions.

What is test retest reliability?

Test-retest reliability assesses the consistency of measurements taken by a test over time. It involves administering the same test to the same participants at two different points in time and comparing the results. A high correlation between the scores indicates that the test produces stable and consistent results over time.

How to improve reliability of an experiment?

• Standardise procedures and instructions.
• Use consistent and precise measurement tools.
• Train observers or raters to reduce subjective judgments.
• Increase sample size to reduce random errors.
• Conduct pilot studies to refine methods.
• Repeat measurements or use multiple methods.
• Address potential sources of variability.

What is the difference between reliability and validity?

Reliability refers to the consistency and repeatability of measurements, ensuring results are stable over time. Validity indicates how well an instrument measures what it’s intended to measure, ensuring accuracy and relevance. While a test can be reliable without being valid, a valid test must inherently be reliable. Both are essential for credible research.

Are interviews reliable and valid?

Interviews can be both reliable and valid, but they are susceptible to biases. The reliability and validity depend on the design, structure, and execution of the interview. Structured interviews with standardised questions improve reliability. Validity is enhanced when questions accurately capture the intended construct and when interviewer biases are minimised.

Are IQ tests valid and reliable?

IQ tests are generally considered reliable, producing consistent scores over time. Their validity, however, is a subject of debate. While they effectively measure certain cognitive skills, whether they capture the entirety of “intelligence” or predict success in all life areas is contested. Cultural bias and over-reliance on tests are also concerns.

Are questionnaires reliable and valid?

Questionnaires can be both reliable and valid if well-designed. Reliability is achieved when they produce consistent results over time or across similar populations. Validity is ensured when questions accurately measure the intended construct. However, factors like poorly phrased questions, respondent bias, and lack of standardisation can compromise their reliability and validity.

You May Also Like

This article presents the key advantages and disadvantages of secondary research so you can select the most appropriate research approach for your study.

A survey includes questions relevant to the research topic. The participants are selected, and the questionnaire is distributed to collect the data.

What are the different types of research you can use in your dissertation? Here are some guidelines to help you choose a research strategy that would make your research more credible.

USEFUL LINKS

LEARNING RESOURCES

COMPANY DETAILS

• How It Works

Validity & Reliability In Research

A Plain-Language Explanation (With Examples)

By: Derek Jansen (MBA) | Expert Reviewer: Kerryn Warren (PhD) | September 2023

Validity and reliability are two related but distinctly different concepts within research. Understanding what they are and how to achieve them is critically important to any research project. In this post, we’ll unpack these two concepts as simply as possible.

This post is based on our popular online course, Research Methodology Bootcamp . In the course, we unpack the basics of methodology  using straightfoward language and loads of examples. If you’re new to academic research, you definitely want to use this link to get 50% off the course (limited-time offer).

Overview: Validity & Reliability

• The big picture
• Validity 101
• Reliability 101 
• Key takeaways

First, The Basics…

First, let’s start with a big-picture view and then we can zoom in to the finer details.

Validity and reliability are two incredibly important concepts in research, especially within the social sciences. Both validity and reliability have to do with the measurement of variables and/or constructs – for example, job satisfaction, intelligence, productivity, etc. When undertaking research, you’ll often want to measure these types of constructs and variables and, at the simplest level, validity and reliability are about ensuring the quality and accuracy of those measurements .

As you can probably imagine, if your measurements aren’t accurate or there are quality issues at play when you’re collecting your data, your entire study will be at risk. Therefore, validity and reliability are very important concepts to understand (and to get right). So, let’s unpack each of them.

What Is Validity?

In simple terms, validity (also called “construct validity”) is all about whether a research instrument accurately measures what it’s supposed to measure .

For example, let’s say you have a set of Likert scales that are supposed to quantify someone’s level of overall job satisfaction. If this set of scales focused purely on only one dimension of job satisfaction, say pay satisfaction, this would not be a valid measurement, as it only captures one aspect of the multidimensional construct. In other words, pay satisfaction alone is only one contributing factor toward overall job satisfaction, and therefore it’s not a valid way to measure someone’s job satisfaction.

Oftentimes in quantitative studies, the way in which the researcher or survey designer interprets a question or statement can differ from how the study participants interpret it . Given that respondents don’t have the opportunity to ask clarifying questions when taking a survey, it’s easy for these sorts of misunderstandings to crop up. Naturally, if the respondents are interpreting the question in the wrong way, the data they provide will be pretty useless . Therefore, ensuring that a study’s measurement instruments are valid – in other words, that they are measuring what they intend to measure – is incredibly important.

There are various types of validity and we’re not going to go down that rabbit hole in this post, but it’s worth quickly highlighting the importance of making sure that your research instrument is tightly aligned with the theoretical construct you’re trying to measure .  In other words, you need to pay careful attention to how the key theories within your study define the thing you’re trying to measure – and then make sure that your survey presents it in the same way.

For example, sticking with the “job satisfaction” construct we looked at earlier, you’d need to clearly define what you mean by job satisfaction within your study (and this definition would of course need to be underpinned by the relevant theory). You’d then need to make sure that your chosen definition is reflected in the types of questions or scales you’re using in your survey . Simply put, you need to make sure that your survey respondents are perceiving your key constructs in the same way you are. Or, even if they’re not, that your measurement instrument is capturing the necessary information that reflects your definition of the construct at hand.

If all of this talk about constructs sounds a bit fluffy, be sure to check out Research Methodology Bootcamp , which will provide you with a rock-solid foundational understanding of all things methodology-related. Remember, you can take advantage of our 60% discount offer using this link.

Need a helping hand?

What Is Reliability?

As with validity, reliability is an attribute of a measurement instrument – for example, a survey, a weight scale or even a blood pressure monitor. But while validity is concerned with whether the instrument is measuring the “thing” it’s supposed to be measuring, reliability is concerned with consistency and stability . In other words, reliability reflects the degree to which a measurement instrument produces consistent results when applied repeatedly to the same phenomenon , under the same conditions .

As you can probably imagine, a measurement instrument that achieves a high level of consistency is naturally more dependable (or reliable) than one that doesn’t – in other words, it can be trusted to provide consistent measurements . And that, of course, is what you want when undertaking empirical research. If you think about it within a more domestic context, just imagine if you found that your bathroom scale gave you a different number every time you hopped on and off of it – you wouldn’t feel too confident in its ability to measure the variable that is your body weight 🙂

It’s worth mentioning that reliability also extends to the person using the measurement instrument . For example, if two researchers use the same instrument (let’s say a measuring tape) and they get different measurements, there’s likely an issue in terms of how one (or both) of them are using the measuring tape. So, when you think about reliability, consider both the instrument and the researcher as part of the equation.

As with validity, there are various types of reliability and various tests that can be used to assess the reliability of an instrument. A popular one that you’ll likely come across for survey instruments is Cronbach’s alpha , which is a statistical measure that quantifies the degree to which items within an instrument (for example, a set of Likert scales) measure the same underlying construct . In other words, Cronbach’s alpha indicates how closely related the items are and whether they consistently capture the same concept . 

Recap: Key Takeaways

Alright, let’s quickly recap to cement your understanding of validity and reliability:

• Validity is concerned with whether an instrument (e.g., a set of Likert scales) is measuring what it’s supposed to measure
• Reliability is concerned with whether that measurement is consistent and stable when measuring the same phenomenon under the same conditions.

In short, validity and reliability are both essential to ensuring that your data collection efforts deliver high-quality, accurate data that help you answer your research questions . So, be sure to always pay careful attention to the validity and reliability of your measurement instruments when collecting and analysing data. As the adage goes, “rubbish in, rubbish out” – make sure that your data inputs are rock-solid.

Psst… there’s more!

This post is an extract from our bestselling short course, Methodology Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

THE MATERIAL IS WONDERFUL AND BENEFICIAL TO ALL STUDENTS.

THE MATERIAL IS WONDERFUL AND BENEFICIAL TO ALL STUDENTS AND I HAVE GREATLY BENEFITED FROM THE CONTENT.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case NPS+ Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Reliability vs. Validity in Research: Types & Examples

When it comes to research, getting things right is crucial. That’s where the concepts of “Reliability vs Validity in Research” come in. 

Imagine it like a balancing act – making sure your measurements are consistent and accurate at the same time. This is where test-retest reliability, having different researchers check things, and keeping things consistent within your research plays a big role. 

As we dive into this topic, we’ll uncover the differences between reliability and validity, see how they work together, and learn how to use them effectively.

Understanding Reliability vs. Validity in Research

When it comes to collecting data and conducting research, two crucial concepts stand out: reliability and validity. 

These pillars uphold the integrity of research findings, ensuring that the data collected and the conclusions drawn are both meaningful and trustworthy. Let’s dive into the heart of the concepts, reliability, and validity, to comprehend their significance in the realm of research truly.

What is reliability?

Reliability refers to the consistency and dependability of the data collection process. It’s like having a steady hand that produces the same result each time it reaches for a task. 

In the research context, reliability is all about ensuring that if you were to repeat the same study using the same reliable measurement technique, you’d end up with the same results. It’s like having multiple researchers independently conduct the same experiment and getting outcomes that align perfectly.

Imagine you’re using a thermometer to measure the temperature of the water. You have a reliable measurement if you dip the thermometer into the water multiple times and get the same reading each time. This tells you that your method and measurement technique consistently produce the same results, whether it’s you or another researcher performing the measurement.

What is validity?

On the other hand, validity refers to the accuracy and meaningfulness of your data. It’s like ensuring that the puzzle pieces you’re putting together actually form the intended picture. When you have validity, you know that your method and measurement technique are consistent and capable of producing results aligned with reality.

Think of it this way; Imagine you’re conducting a test that claims to measure a specific trait, like problem-solving ability. If the test consistently produces results that accurately reflect participants’ problem-solving skills, then the test has high validity. In this case, the test produces accurate results that truly correspond to the trait it aims to measure.

In essence, while reliability assures you that your data collection process is like a well-oiled machine producing the same results, validity steps in to ensure that these results are not only consistent but also relevantly accurate. 

Together, these concepts provide researchers with the tools to conduct research that stands on a solid foundation of dependable methods and meaningful insights.

Types of Reliability

Let’s explore the various types of reliability that researchers consider to ensure their work stands on solid ground.

High test-retest reliability

Test-retest reliability involves assessing the consistency of measurements over time. It’s like taking the same measurement or test twice – once and then again after a certain period. If the results align closely, it indicates that the measurement is reliable over time. Think of it as capturing the essence of stability. 

Inter-rater reliability

When multiple researchers or observers are part of the equation, interrater reliability comes into play. This type of reliability assesses the level of agreement between different observers when evaluating the same phenomenon. It’s like ensuring that different pairs of eyes perceive things in a similar way. 

Internal reliability

Internal consistency dives into the harmony among different items within a measurement tool aiming to assess the same concept. This often comes into play in surveys or questionnaires, where participants respond to various items related to a single construct. If the responses to these items consistently reflect the same underlying concept, the measurement is said to have high internal consistency. 

Types of validity

Let’s explore the various types of validity that researchers consider to ensure their work stands on solid ground.

Content validity

It delves into whether a measurement truly captures all dimensions of the concept it intends to measure. It’s about making sure your measurement tool covers all relevant aspects comprehensively. 

Imagine designing a test to assess students’ understanding of a history chapter. It exhibits high content validity if the test includes questions about key events, dates, and causes. However, if it focuses solely on dates and omits causation, its content validity might be questionable.

Construct validity

It assesses how well a measurement aligns with established theories and concepts. It’s like ensuring that your measurement is a true representation of the abstract construct you’re trying to capture. 

Criterion validity

Criterion validity examines how well your measurement corresponds to other established measurements of the same concept. It’s about making sure your measurement accurately predicts or correlates with external criteria.

Differences between reliability and validity in research

Let’s delve into the differences between reliability and validity in research.

While both reliability and validity contribute to trustworthy research, they address distinct aspects. Reliability ensures consistent results, while validity ensures accurate and relevant results that reflect the true nature of the measured concept.

Example of Reliability and Validity in Research

In this section, we’ll explore instances that highlight the differences between reliability and validity and how they play a crucial role in ensuring the credibility of research findings.

Example of reliability

Imagine you are studying the reliability of a smartphone’s battery life measurement. To collect data, you fully charge the phone and measure the battery life three times in the same controlled environment—same apps running, same brightness level, and same usage patterns. 

If the measurements consistently show a similar battery life duration each time you repeat the test, it indicates that your measurement method is reliable. The consistent results under the same conditions assure you that the battery life measurement can be trusted to provide dependable information about the phone’s performance.

Example of validity

Researchers collect data from a group of participants in a study aiming to assess the validity of a newly developed stress questionnaire. To ensure validity, they compare the scores obtained from the stress questionnaire with the participants’ actual stress levels measured using physiological indicators such as heart rate variability and cortisol levels. 

If participants’ scores correlate strongly with their physiological stress levels, the questionnaire is valid. This means the questionnaire accurately measures participants’ stress levels, and its results correspond to real variations in their physiological responses to stress. 

Validity assessed through the correlation between questionnaire scores and physiological measures ensures that the questionnaire is effectively measuring what it claims to measure participants’ stress levels.

In the world of research, differentiating between reliability and validity is crucial. Reliability ensures consistent results, while validity confirms accurate measurements. Using tools like QuestionPro enhances data collection for both reliability and validity. For instance, measuring self-esteem over time showcases reliability, and aligning questions with theories demonstrates validity. 

QuestionPro empowers researchers to achieve reliable and valid results through its robust features, facilitating credible research outcomes. Contact QuestionPro to create a free account or learn more!

LEARN MORE         FREE TRIAL

MORE LIKE THIS

What Are My Employees Really Thinking? The Power of Open-ended Survey Analysis

May 24, 2024

I Am Disconnected – Tuesday CX Thoughts

May 21, 2024

20 Best Customer Success Tools of 2024

May 20, 2024

AI-Based Services Buying Guide for Market Research (based on ESOMAR’s 20 Questions) 

Other categories.

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Uncategorized
• Video Learning Series
• What’s Coming Up
• Workforce Intelligence

• Reliability vs Validity in Research: Types & Examples

In everyday life, we probably use reliability to describe how something is valid. However, in research and testing, reliability and validity are not the same things.

When it comes to data analysis, reliability refers to how easily replicable an outcome is. For example, if you measure a cup of rice three times, and you get the same result each time, that result is reliable.

The validity, on the other hand, refers to the measurement’s accuracy. This means that if the standard weight for a cup of rice is 5 grams, and you measure a cup of rice, it should be 5 grams.

So, while reliability and validity are intertwined, they are not synonymous. If one of the measurement parameters, such as your scale, is distorted, the results will be consistent but invalid.

Data must be consistent and accurate to be used to draw useful conclusions. In this article, we’ll look at how to assess data reliability and validity, as well as how to apply it.

Read: Internal Validity in Research: Definition, Threats, Examples

What is Reliability?

When a measurement is consistent it’s reliable. But of course, reliability doesn’t mean your outcome will be the same, it just means it will be in the same range. 

For example, if you scored 95% on a test the first time and the next you score, 96%, your results are reliable.  So, even if there is a minor difference in the outcomes, as long as it is within the error margin, your results are reliable.

Reliability allows you to assess the degree of consistency in your results. So, if you’re getting similar results, reliability provides an answer to the question of how similar your results are.

What is Validity?

A measurement or test is valid when it correlates with the expected result. It examines the accuracy of your result.

Here’s where things get tricky: to establish the validity of a test, the results must be consistent. Looking at most experiments (especially physical measurements), the standard value that establishes the accuracy of a measurement is the outcome of repeating the test to obtain a consistent result.

Read: What is Participant Bias? How to Detect & Avoid It

For example, before I can conclude that all 12-inch rulers are one foot, I must repeat the experiment several times and obtain very similar results, indicating that 12-inch rulers are indeed one foot.

Most scientific experiments are inextricably linked in terms of validity and reliability. For example, if you’re measuring distance or depth, valid answers are likely to be reliable.

But for social experiences, one isn’t the indication of the other. For example, most people believe that people that wear glasses are smart. 

Of course, I’ll find examples of people who wear glasses and have high IQs (reliability), but the truth is that most people who wear glasses simply need their vision to be better (validity). 

So reliable answers aren’t always correct but valid answers are always reliable.

How Are Reliability and Validity Assessed?

When assessing reliability, we want to know if the measurement can be replicated. Of course, we’d have to change some variables to ensure that this test holds, the most important of which are time, items, and observers.

If the main factor you change when performing a reliability test is time, you’re performing a test-retest reliability assessment.

Read: What is Publication Bias? (How to Detect & Avoid It)

However, if you are changing items, you are performing an internal consistency assessment. It means you’re measuring multiple items with a single instrument.

Finally, if you’re measuring the same item with the same instrument but using different observers or judges, you’re performing an inter-rater reliability test.

Assessing Validity

Evaluating validity can be more tedious than reliability. With reliability, you’re attempting to demonstrate that your results are consistent, whereas, with validity, you want to prove the correctness of your outcome.

Although validity is mainly categorized under two sections (internal and external), there are more than fifteen ways to check the validity of a test. In this article, we’ll be covering four.

First, content validity, measures whether the test covers all the content it needs to provide the outcome you’re expecting. 

Suppose I wanted to test the hypothesis that 90% of Generation Z uses social media polls for surveys while 90% of millennials use forms. I’d need a sample size that accounts for how Gen Z and millennials gather information.

Next, criterion validity is when you compare your results to what you’re supposed to get based on a chosen criteria. There are two ways these could be measured, predictive or concurrent validity.

Read: Survey Errors To Avoid: Types, Sources, Examples, Mitigation

Following that, we have face validity . It’s how we anticipate a test to be. For instance, when answering a customer service survey, I’d expect to be asked about how I feel about the service provided.

Lastly, construct-related validity . This is a little more complicated, but it helps to show how the validity of research is based on different findings.

As a result, it provides information that either proves or disproves that certain things are related.

Types of Reliability

We have three main types of reliability assessment and here’s how they work:

1) Test-retest Reliability

This assessment refers to the consistency of outcomes over time. Testing reliability over time does not imply changing the amount of time it takes to conduct an experiment; rather, it means repeating the experiment multiple times in a short time.

For example, if I measure the length of my hair today, and tomorrow, I’ll most likely get the same result each time. 

A short period is relative in terms of reliability; two days for measuring hair length is considered short. But that’s far too long to test how quickly water dries on the sand.

A test-retest correlation is used to compare the consistency of your results. This is typically a scatter plot that shows how similar your values are between the two experiments.

If your answers are reliable, your scatter plots will most likely have a lot of overlapping points, but if they aren’t, the points (values) will be spread across the graph.

Read: Sampling Bias: Definition, Types + [Examples]

2) Internal Consistency

It’s also known as internal reliability. It refers to the consistency of results for various items when measured on the same scale.

This is particularly important in social science research, such as surveys, because it helps determine the consistency of people’s responses when asked the same questions.

Most introverts, for example, would say they enjoy spending time alone and having few friends. However, if some introverts claim that they either do not want time alone or prefer to be surrounded by many friends, it doesn’t add up.

These people who claim to be introverts or one this factor isn’t a reliable way of measuring introversion.

Internal reliability helps you prove the consistency of a test by varying factors. It’s a little tough to measure quantitatively but you could use the split-half correlation .

The split-half correlation simply means dividing the factors used to measure the underlying construct into two and plotting them against each other in the form of a scatter plot.

Introverts, for example, are assessed on their need for alone time as well as their desire to have as few friends as possible. If this plot is dispersed, likely, one of the traits does not indicate introversion.

3) Inter-Rater Reliability

This method of measuring reliability helps prevent personal bias. Inter-rater reliability assessment helps judge outcomes from the different perspectives of multiple observers.

A good example is if you ordered a meal and found it delicious. You could be biased in your judgment for several reasons, perception of the meal, your mood, and so on.

But it’s highly unlikely that six more people would agree that the meal is delicious if it isn’t. Another factor that could lead to bias is expertise. Professional dancers, for example, would perceive dance moves differently than non-professionals. 

Read: What is Experimenter Bias? Definition, Types & Mitigation

So, if a person dances and records it, and both groups (professional and unprofessional dancers) rate the video, there is a high likelihood of a significant difference in their ratings.

But if they both agree that the person is a great dancer, despite their opposing viewpoints, the person is likely a great dancer.

Types of Validity

Researchers use validity to determine whether a measurement is accurate or not. The accuracy of measurement is usually determined by comparing it to the standard value.

When a measurement is consistent over time and has high internal consistency, it increases the likelihood that it is valid.

1) Content Validity

This refers to determining validity by evaluating what is being measured. So content validity tests if your research is measuring everything it should to produce an accurate result.

For example, if I were to measure what causes hair loss in women. I’d have to consider things like postpartum hair loss, alopecia, hair manipulation, dryness, and so on.

By omitting any of these critical factors, you risk significantly reducing the validity of your research because you won’t be covering everything necessary to make an accurate deduction. 

Read: Data Cleaning: 7 Techniques + Steps to Cleanse Data

For example, a certain woman is losing her hair due to postpartum hair loss, excessive manipulation, and dryness, but in my research, I only look at postpartum hair loss. My research will show that she has postpartum hair loss, which isn’t accurate.

Yes, my conclusion is correct, but it does not fully account for the reasons why this woman is losing her hair.

2) Criterion Validity

This measures how well your measurement correlates with the variables you want to compare it with to get your result. The two main classes of criterion validity are predictive and concurrent.

3) Predictive validity

It helps predict future outcomes based on the data you have. For example, if a large number of students performed exceptionally well in a test, you can use this to predict that they understood the concept on which the test was based and will perform well in their exams.

4) Concurrent validity

On the other hand, involves testing with different variables at the same time. For example, setting up a literature test for your students on two different books and assessing them at the same time.

You’re measuring your students’ literature proficiency with these two books. If your students truly understood the subject, they should be able to correctly answer questions about both books.

5) Face Validity

Quantifying face validity might be a bit difficult because you are measuring the perception validity, not the validity itself. So, face validity is concerned with whether the method used for measurement will produce accurate results rather than the measurement itself.

If the method used for measurement doesn’t appear to test the accuracy of a measurement, its face validity is low.

Here’s an example: less than 40% of men over the age of 20 in Texas, USA, are at least 6 feet tall. The most logical approach would be to collect height data from men over the age of twenty in Texas, USA.

However, asking men over the age of 20 what their favorite meal is to determine their height is pretty bizarre. The method I am using to assess the validity of my research is quite questionable because it lacks correlation to what I want to measure.

6) Construct-Related Validity

Construct-related validity assesses the accuracy of your research by collecting multiple pieces of evidence. It helps determine the validity of your results by comparing them to evidence that supports or refutes your measurement.

7) Convergent validity

If you’re assessing evidence that strongly correlates with the concept, that’s convergent validity . 

8) Discriminant validity

Examines the validity of your research by determining what not to base it on. You are removing elements that are not a strong factor to help validate your research. Being a vegan, for example, does not imply that you are allergic to meat.

How to Ensure Validity and Reliability in Your Research

You need a bulletproof research design to ensure that your research is both valid and reliable. This means that your methods, sample, and even you, the researcher, shouldn’t be biased.

• Ensuring Reliability

To enhance the reliability of your research, you need to apply your measurement method consistently. The chances of reproducing the same results for a test are higher when you maintain the method you’re using to experiment.

For example, you want to determine the reliability of the weight of a bag of chips using a scale. You have to consistently use this scale to measure the bag of chips each time you experiment.

You must also keep the conditions of your research consistent. For instance, if you’re experimenting to see how quickly water dries on sand, you need to consider all of the weather elements that day.

So, if you experimented on a sunny day, the next experiment should also be conducted on a sunny day to obtain a reliable result.

Read: Survey Methods: Definition, Types, and Examples

• Ensuring Validity

There are several ways to determine the validity of your research, and the majority of them require the use of highly specific and high-quality measurement methods.

Before you begin your test, choose the best method for producing the desired results. This method should be pre-existing and proven.

Also, your sample should be very specific. If you’re collecting data on how dogs respond to fear, your results are more likely to be valid if you base them on a specific breed of dog rather than dogs in general.

Validity and reliability are critical for achieving accurate and consistent results in research. While reliability does not always imply validity, validity establishes that a result is reliable. Validity is heavily dependent on previous results (standards), whereas reliability is dependent on the similarity of your results.

Connect to Formplus, Get Started Now - It's Free!

• concurrent validity
• examples of research bias
• predictive reliability
• research analysis
• research assessment
• validity of research
• busayo.longe

You may also like:

How to do a Meta Analysis: Methodology, Pros & Cons

In this article, we’ll go through the concept of meta-analysis, what it can be used for, and how you can use it to improve how you...

Research Bias: Definition, Types + Examples

Simple guide to understanding research bias, types, causes, examples and how to avoid it in surveys

Selection Bias in Research: Types, Examples & Impact

In this article, we’ll discuss the effects of selection bias, how it works, its common effects and the best ways to minimize it.

Simpson’s Paradox & How to Avoid it in Experimental Research

In this article, we are going to look at Simpson’s Paradox from its historical point and later, we’ll consider its effect in...

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

• Privacy Policy

Home » Validity – Types, Examples and Guide

Validity – Types, Examples and Guide

Table of Contents

Definition:

Validity refers to the extent to which a concept, measure, or study accurately represents the intended meaning or reality it is intended to capture. It is a fundamental concept in research and assessment that assesses the soundness and appropriateness of the conclusions, inferences, or interpretations made based on the data or evidence collected.

Research Validity

Research validity refers to the degree to which a study accurately measures or reflects what it claims to measure. In other words, research validity concerns whether the conclusions drawn from a study are based on accurate, reliable and relevant data.

Validity is a concept used in logic and research methodology to assess the strength of an argument or the quality of a research study. It refers to the extent to which a conclusion or result is supported by evidence and reasoning.

How to Ensure Validity in Research

Ensuring validity in research involves several steps and considerations throughout the research process. Here are some key strategies to help maintain research validity:

Clearly Define Research Objectives and Questions

Start by clearly defining your research objectives and formulating specific research questions. This helps focus your study and ensures that you are addressing relevant and meaningful research topics.

Use appropriate research design

Select a research design that aligns with your research objectives and questions. Different types of studies, such as experimental, observational, qualitative, or quantitative, have specific strengths and limitations. Choose the design that best suits your research goals.

Use reliable and valid measurement instruments

If you are measuring variables or constructs, ensure that the measurement instruments you use are reliable and valid. This involves using established and well-tested tools or developing your own instruments through rigorous validation processes.

Ensure a representative sample

When selecting participants or subjects for your study, aim for a sample that is representative of the population you want to generalize to. Consider factors such as age, gender, socioeconomic status, and other relevant demographics to ensure your findings can be generalized appropriately.

Address potential confounding factors

Identify potential confounding variables or biases that could impact your results. Implement strategies such as randomization, matching, or statistical control to minimize the influence of confounding factors and increase internal validity.

Minimize measurement and response biases

Be aware of measurement biases and response biases that can occur during data collection. Use standardized protocols, clear instructions, and trained data collectors to minimize these biases. Employ techniques like blinding or double-blinding in experimental studies to reduce bias.

Conduct appropriate statistical analyses

Ensure that the statistical analyses you employ are appropriate for your research design and data type. Select statistical tests that are relevant to your research questions and use robust analytical techniques to draw accurate conclusions from your data.

Consider external validity

While it may not always be possible to achieve high external validity, be mindful of the generalizability of your findings. Clearly describe your sample and study context to help readers understand the scope and limitations of your research.

Peer review and replication

Submit your research for peer review by experts in your field. Peer review helps identify potential flaws, biases, or methodological issues that can impact validity. Additionally, encourage replication studies by other researchers to validate your findings and enhance the overall reliability of the research.

Transparent reporting

Clearly and transparently report your research methods, procedures, data collection, and analysis techniques. Provide sufficient details for others to evaluate the validity of your study and replicate your work if needed.

Types of Validity

There are several types of validity that researchers consider when designing and evaluating studies. Here are some common types of validity:

Internal Validity

Internal validity relates to the degree to which a study accurately identifies causal relationships between variables. It addresses whether the observed effects can be attributed to the manipulated independent variable rather than confounding factors. Threats to internal validity include selection bias, history effects, maturation of participants, and instrumentation issues.

External Validity

External validity concerns the generalizability of research findings to the broader population or real-world settings. It assesses the extent to which the results can be applied to other individuals, contexts, or timeframes. Factors that can limit external validity include sample characteristics, research settings, and the specific conditions under which the study was conducted.

Construct Validity

Construct validity examines whether a study adequately measures the intended theoretical constructs or concepts. It focuses on the alignment between the operational definitions used in the study and the underlying theoretical constructs. Construct validity can be threatened by issues such as poor measurement tools, inadequate operational definitions, or a lack of clarity in the conceptual framework.

Content Validity

Content validity refers to the degree to which a measurement instrument or test adequately covers the entire range of the construct being measured. It assesses whether the items or questions included in the measurement tool represent the full scope of the construct. Content validity is often evaluated through expert judgment, reviewing the relevance and representativeness of the items.

Criterion Validity

Criterion validity determines the extent to which a measure or test is related to an external criterion or standard. It assesses whether the results obtained from a measurement instrument align with other established measures or outcomes. Criterion validity can be divided into two subtypes: concurrent validity, which examines the relationship between the measure and the criterion at the same time, and predictive validity, which investigates the measure’s ability to predict future outcomes.

Face Validity

Face validity refers to the degree to which a measurement or test appears, on the surface, to measure what it intends to measure. It is a subjective assessment based on whether the items seem relevant and appropriate to the construct being measured. Face validity is often used as an initial evaluation before conducting more rigorous validity assessments.

Importance of Validity

Validity is crucial in research for several reasons:

• Accurate Measurement: Validity ensures that the measurements or observations in a study accurately represent the intended constructs or variables. Without validity, researchers cannot be confident that their results truly reflect the phenomena they are studying. Validity allows researchers to draw accurate conclusions and make meaningful inferences based on their findings.
• Credibility and Trustworthiness: Validity enhances the credibility and trustworthiness of research. When a study demonstrates high validity, it indicates that the researchers have taken appropriate measures to ensure the accuracy and integrity of their work. This strengthens the confidence of other researchers, peers, and the wider scientific community in the study’s results and conclusions.
• Generalizability: Validity helps determine the extent to which research findings can be generalized beyond the specific sample and context of the study. By addressing external validity, researchers can assess whether their results can be applied to other populations, settings, or situations. This information is valuable for making informed decisions, implementing interventions, or developing policies based on research findings.
• Sound Decision-Making: Validity supports informed decision-making in various fields, such as medicine, psychology, education, and social sciences. When validity is established, policymakers, practitioners, and professionals can rely on research findings to guide their actions and interventions. Validity ensures that decisions are based on accurate and trustworthy information, which can lead to better outcomes and more effective practices.
• Avoiding Errors and Bias: Validity helps researchers identify and mitigate potential errors and biases in their studies. By addressing internal validity, researchers can minimize confounding factors and alternative explanations, ensuring that the observed effects are genuinely attributable to the manipulated variables. Validity assessments also highlight measurement errors or shortcomings, enabling researchers to improve their measurement tools and procedures.
• Progress of Scientific Knowledge: Validity is essential for the advancement of scientific knowledge. Valid research contributes to the accumulation of reliable and valid evidence, which forms the foundation for building theories, developing models, and refining existing knowledge. Validity allows researchers to build upon previous findings, replicate studies, and establish a cumulative body of knowledge in various disciplines. Without validity, the scientific community would struggle to make meaningful progress and establish a solid understanding of the phenomena under investigation.
• Ethical Considerations: Validity is closely linked to ethical considerations in research. Conducting valid research ensures that participants’ time, effort, and data are not wasted on flawed or invalid studies. It upholds the principle of respect for participants’ autonomy and promotes responsible research practices. Validity is also important when making claims or drawing conclusions that may have real-world implications, as misleading or invalid findings can have adverse effects on individuals, organizations, or society as a whole.

Examples of Validity

Here are some examples of validity in different contexts:

• Example 1: All men are mortal. John is a man. Therefore, John is mortal. This argument is logically valid because the conclusion follows logically from the premises.
• Example 2: If it is raining, then the ground is wet. The ground is wet. Therefore, it is raining. This argument is not logically valid because there could be other reasons for the ground being wet, such as watering the plants.
• Example 1: In a study examining the relationship between caffeine consumption and alertness, the researchers use established measures of both variables, ensuring that they are accurately capturing the concepts they intend to measure. This demonstrates construct validity.
• Example 2: A researcher develops a new questionnaire to measure anxiety levels. They administer the questionnaire to a group of participants and find that it correlates highly with other established anxiety measures. This indicates good construct validity for the new questionnaire.
• Example 1: A study on the effects of a particular teaching method is conducted in a controlled laboratory setting. The findings of the study may lack external validity because the conditions in the lab may not accurately reflect real-world classroom settings.
• Example 2: A research study on the effects of a new medication includes participants from diverse backgrounds and age groups, increasing the external validity of the findings to a broader population.
• Example 1: In an experiment, a researcher manipulates the independent variable (e.g., a new drug) and controls for other variables to ensure that any observed effects on the dependent variable (e.g., symptom reduction) are indeed due to the manipulation. This establishes internal validity.
• Example 2: A researcher conducts a study examining the relationship between exercise and mood by administering questionnaires to participants. However, the study lacks internal validity because it does not control for other potential factors that could influence mood, such as diet or stress levels.
• Example 1: A teacher develops a new test to assess students’ knowledge of a particular subject. The items on the test appear to be relevant to the topic at hand and align with what one would expect to find on such a test. This suggests face validity, as the test appears to measure what it intends to measure.
• Example 2: A company develops a new customer satisfaction survey. The questions included in the survey seem to address key aspects of the customer experience and capture the relevant information. This indicates face validity, as the survey seems appropriate for assessing customer satisfaction.
• Example 1: A team of experts reviews a comprehensive curriculum for a high school biology course. They evaluate the curriculum to ensure that it covers all the essential topics and concepts necessary for students to gain a thorough understanding of biology. This demonstrates content validity, as the curriculum is representative of the domain it intends to cover.
• Example 2: A researcher develops a questionnaire to assess career satisfaction. The questions in the questionnaire encompass various dimensions of job satisfaction, such as salary, work-life balance, and career growth. This indicates content validity, as the questionnaire adequately represents the different aspects of career satisfaction.
• Example 1: A company wants to evaluate the effectiveness of a new employee selection test. They administer the test to a group of job applicants and later assess the job performance of those who were hired. If there is a strong correlation between the test scores and subsequent job performance, it suggests criterion validity, indicating that the test is predictive of job success.
• Example 2: A researcher wants to determine if a new medical diagnostic tool accurately identifies a specific disease. They compare the results of the diagnostic tool with the gold standard diagnostic method and find a high level of agreement. This demonstrates criterion validity, indicating that the new tool is valid in accurately diagnosing the disease.

Where to Write About Validity in A Thesis

In a thesis, discussions related to validity are typically included in the methodology and results sections. Here are some specific places where you can address validity within your thesis:

Research Design and Methodology

In the methodology section, provide a clear and detailed description of the measures, instruments, or data collection methods used in your study. Discuss the steps taken to establish or assess the validity of these measures. Explain the rationale behind the selection of specific validity types relevant to your study, such as content validity, criterion validity, or construct validity. Discuss any modifications or adaptations made to existing measures and their potential impact on validity.

Measurement Procedures

In the methodology section, elaborate on the procedures implemented to ensure the validity of measurements. Describe how potential biases or confounding factors were addressed, controlled, or accounted for to enhance internal validity. Provide details on how you ensured that the measurement process accurately captures the intended constructs or variables of interest.

Data Collection

In the methodology section, discuss the steps taken to collect data and ensure data validity. Explain any measures implemented to minimize errors or biases during data collection, such as training of data collectors, standardized protocols, or quality control procedures. Address any potential limitations or threats to validity related to the data collection process.

Data Analysis and Results

In the results section, present the analysis and findings related to validity. Report any statistical tests, correlations, or other measures used to assess validity. Provide interpretations and explanations of the results obtained. Discuss the implications of the validity findings for the overall reliability and credibility of your study.

Limitations and Future Directions

In the discussion or conclusion section, reflect on the limitations of your study, including limitations related to validity. Acknowledge any potential threats or weaknesses to validity that you encountered during your research. Discuss how these limitations may have influenced the interpretation of your findings and suggest avenues for future research that could address these validity concerns.

Applications of Validity

Validity is applicable in various areas and contexts where research and measurement play a role. Here are some common applications of validity:

Psychological and Behavioral Research

Validity is crucial in psychology and behavioral research to ensure that measurement instruments accurately capture constructs such as personality traits, intelligence, attitudes, emotions, or psychological disorders. Validity assessments help researchers determine if their measures are truly measuring the intended psychological constructs and if the results can be generalized to broader populations or real-world settings.

Educational Assessment

Validity is essential in educational assessment to determine if tests, exams, or assessments accurately measure students’ knowledge, skills, or abilities. It ensures that the assessment aligns with the educational objectives and provides reliable information about student performance. Validity assessments help identify if the assessment is valid for all students, regardless of their demographic characteristics, language proficiency, or cultural background.

Program Evaluation

Validity plays a crucial role in program evaluation, where researchers assess the effectiveness and impact of interventions, policies, or programs. By establishing validity, evaluators can determine if the observed outcomes are genuinely attributable to the program being evaluated rather than extraneous factors. Validity assessments also help ensure that the evaluation findings are applicable to different populations, contexts, or timeframes.

Medical and Health Research

Validity is essential in medical and health research to ensure the accuracy and reliability of diagnostic tools, measurement instruments, and clinical assessments. Validity assessments help determine if a measurement accurately identifies the presence or absence of a medical condition, measures the effectiveness of a treatment, or predicts patient outcomes. Validity is crucial for establishing evidence-based medicine and informing medical decision-making.

Social Science Research

Validity is relevant in various social science disciplines, including sociology, anthropology, economics, and political science. Researchers use validity to ensure that their measures and methods accurately capture social phenomena, such as social attitudes, behaviors, social structures, or economic indicators. Validity assessments support the reliability and credibility of social science research findings.

Market Research and Surveys

Validity is important in market research and survey studies to ensure that the survey questions effectively measure consumer preferences, buying behaviors, or attitudes towards products or services. Validity assessments help researchers determine if the survey instrument is accurately capturing the desired information and if the results can be generalized to the target population.

Limitations of Validity

Here are some limitations of validity:

• Construct Validity: Limitations of construct validity include the potential for measurement error, inadequate operational definitions of constructs, or the failure to capture all aspects of a complex construct.
• Internal Validity: Limitations of internal validity may arise from confounding variables, selection bias, or the presence of extraneous factors that could influence the study outcomes, making it difficult to attribute causality accurately.
• External Validity: Limitations of external validity can occur when the study sample does not represent the broader population, when the research setting differs significantly from real-world conditions, or when the study lacks ecological validity, i.e., the findings do not reflect real-world complexities.
• Measurement Validity: Limitations of measurement validity can arise from measurement error, inadequately designed or flawed measurement scales, or limitations inherent in self-report measures, such as social desirability bias or recall bias.
• Statistical Conclusion Validity: Limitations in statistical conclusion validity can occur due to sampling errors, inadequate sample sizes, or improper statistical analysis techniques, leading to incorrect conclusions or generalizations.
• Temporal Validity: Limitations of temporal validity arise when the study results become outdated due to changes in the studied phenomena, interventions, or contextual factors.
• Researcher Bias: Researcher bias can affect the validity of a study. Biases can emerge through the researcher’s subjective interpretation, influence of personal beliefs, or preconceived notions, leading to unintentional distortion of findings or failure to consider alternative explanations.
• Ethical Validity: Limitations can arise if the study design or methods involve ethical concerns, such as the use of deceptive practices, inadequate informed consent, or potential harm to participants.

Also see  Reliability Vs Validity

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Alternate Forms Reliability – Methods, Examples...

Construct Validity – Types, Threats and Examples

Internal Validity – Threats, Examples and Guide

Reliability Vs Validity

Internal Consistency Reliability – Methods...

Split-Half Reliability – Methods, Examples and...

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons

Margin Size

• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

5.13: The Reliability and Validity of Research

• Last updated
• Save as PDF
• Page ID 59853

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)

( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

\( \newcommand{\Span}{\mathrm{span}}\)

\( \newcommand{\id}{\mathrm{id}}\)

\( \newcommand{\kernel}{\mathrm{null}\,}\)

\( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\)

\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\)

\( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)

\( \newcommand{\vectorA}[1]{\vec{#1}}      % arrow\)

\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}}      % arrow\)

\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vectorC}[1]{\textbf{#1}} \)

\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)

\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)

\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)

Learning Objectives

• Define reliability and validity

Interpreting Experimental Findings

Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful differences between the two groups. A statistical analysis determines how likely any difference found is due to chance (and thus not meaningful). In psychology, group differences are considered meaningful, or significant, if the odds that these differences occurred by chance alone are 5 percent or less. Stated another way, if we repeated this experiment 100 times, we would expect to find the same results at least 95 times out of 100.

The greatest strength of experiments is the ability to assert that any significant differences in the findings are caused by the independent variable. This occurs because random selection, random assignment, and a design that limits the effects of both experimenter bias and participant expectancy should create groups that are similar in composition and treatment. Therefore, any difference between the groups is attributable to the independent variable, and now we can finally make a causal statement. If we find that watching a violent television program results in more violent behavior than watching a nonviolent program, we can safely say that watching violent television programs causes an increase in the display of violent behavior.

Reporting Research

When psychologists complete a research project, they generally want to share their findings with other scientists. The American Psychological Association (APA) publishes a manual detailing how to write a paper for submission to scientific journals. Unlike an article that might be published in a magazine like Psychology Today, which targets a general audience with an interest in psychology, scientific journals generally publish peer-reviewed journal articles aimed at an audience of professionals and scholars who are actively involved in research themselves.

Link to Learning

The Online Writing Lab (OWL) at Purdue University can walk you through the APA writing guidelines.

A peer-reviewed journal article is read by several other scientists (generally anonymously) with expertise in the subject matter. These peer reviewers provide feedback—to both the author and the journal editor—regarding the quality of the draft. Peer reviewers look for a strong rationale for the research being described, a clear description of how the research was conducted, and evidence that the research was conducted in an ethical manner. They also look for flaws in the study’s design, methods, and statistical analyses. They check that the conclusions drawn by the authors seem reasonable given the observations made during the research. Peer reviewers also comment on how valuable the research is in advancing the discipline’s knowledge. This helps prevent unnecessary duplication of research findings in the scientific literature and, to some extent, ensures that each research article provides new information. Ultimately, the journal editor will compile all of the peer reviewer feedback and determine whether the article will be published in its current state (a rare occurrence), published with revisions, or not accepted for publication.

Peer review provides some degree of quality control for psychological research. Poorly conceived or executed studies can be weeded out, and even well-designed research can be improved by the revisions suggested. Peer review also ensures that the research is described clearly enough to allow other scientists to replicate it, meaning they can repeat the experiment using different samples to determine reliability. Sometimes replications involve additional measures that expand on the original finding. In any case, each replication serves to provide more evidence to support the original research findings. Successful replications of published research make scientists more apt to adopt those findings, while repeated failures tend to cast doubt on the legitimacy of the original article and lead scientists to look elsewhere. For example, it would be a major advancement in the medical field if a published study indicated that taking a new drug helped individuals achieve a healthy weight without changing their diet. But if other scientists could not replicate the results, the original study’s claims would be questioned.

Dig Deeper: The Vaccine-Autism Myth and the Retraction of Published Studies

Some scientists have claimed that routine childhood vaccines cause some children to develop autism, and, in fact, several peer-reviewed publications published research making these claims. Since the initial reports, large-scale epidemiological research has suggested that vaccinations are not responsible for causing autism and that it is much safer to have your child vaccinated than not. Furthermore, several of the original studies making this claim have since been retracted.

A published piece of work can be rescinded when data is called into question because of falsification, fabrication, or serious research design problems. Once rescinded, the scientific community is informed that there are serious problems with the original publication. Retractions can be initiated by the researcher who led the study, by research collaborators, by the institution that employed the researcher, or by the editorial board of the journal in which the article was originally published. In the vaccine-autism case, the retraction was made because of a significant conflict of interest in which the leading researcher had a financial interest in establishing a link between childhood vaccines and autism (Offit, 2008). Unfortunately, the initial studies received so much media attention that many parents around the world became hesitant to have their children vaccinated (Figure 1). For more information about how the vaccine/autism story unfolded, as well as the repercussions of this story, take a look at Paul Offit’s book, Autism’s False Prophets: Bad Science, Risky Medicine, and the Search for a Cure.

Reliability and Validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways. Unfortunately, being consistent in measurement does not necessarily mean that you have measured something correctly. This is where validity comes into play. Validity refers to the extent to which a given instrument or tool accurately measures what it’s supposed to measure. While any valid measure is by necessity reliable, the reverse is not necessarily true. Researchers strive to use instruments that are both highly reliable and valid.

Query \(\PageIndex{1}\)

Everyday Connection: How Valid Is the SAT?

Standardized tests like the SAT are supposed to measure an individual’s aptitude for a college education, but how reliable and valid are such tests? Research conducted by the College Board suggests that scores on the SAT have high predictive validity for first-year college students’ GPA (Kobrin, Patterson, Shaw, Mattern, & Barbuti, 2008). In this context, predictive validity refers to the test’s ability to effectively predict the GPA of college freshmen. Given that many institutions of higher education require the SAT for admission, this high degree of predictive validity might be comforting.

However, the emphasis placed on SAT scores in college admissions has generated some controversy on a number of fronts. For one, some researchers assert that the SAT is a biased test that places minority students at a disadvantage and unfairly reduces the likelihood of being admitted into a college (Santelices & Wilson, 2010). Additionally, some research has suggested that the predictive validity of the SAT is grossly exaggerated in how well it is able to predict the GPA of first-year college students. In fact, it has been suggested that the SAT’s predictive validity may be overestimated by as much as 150% (Rothstein, 2004). Many institutions of higher education are beginning to consider de-emphasizing the significance of SAT scores in making admission decisions (Rimer, 2008).

Recent examples of high profile cheating scandals both domestically and abroad have only increased the scrutiny being placed on these types of tests, and as of March 2019, more than 1000 institutions of higher education have either relaxed or eliminated the requirements for SAT or ACT testing for admissions (Strauss, 2019, March 19).

Query \(\PageIndex{2}\)

Query \(\PageIndex{3}\)

reliability:  consistency and reproducibility of a given result

Licenses and Attributions

CC licensed content, Shared previously

• Analyzing Findings. Authored by : OpenStax College. Located at : http://cnx.org/contents/[email protected]:mfArybye@7/Analyzing-Findings . License : CC BY: Attribution . License Terms : Download for free at http://cnx.org/contents/[email protected]

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

5.2 Reliability and Validity of Measurement

Learning objectives.

• Define reliability, including the different types and how they are assessed.
• Define validity, including the different types and how they are assessed.
• Describe the kinds of evidence that would be relevant to assessing the reliability and validity of a particular measure.

Again, measurement involves assigning scores to individuals so that they represent some characteristic of the individuals. But how do researchers know that the scores actually represent the characteristic, especially when it is a construct like intelligence, self-esteem, depression, or working memory capacity? The answer is that they conduct research using the measure to confirm that the scores make sense based on their understanding of the construct being measured. This is an extremely important point. Psychologists do not simply assume that their measures work. Instead, they collect data to demonstrate that they work. If their research does not demonstrate that a measure works, they stop using it.

As an informal example, imagine that you have been dieting for a month. Your clothes seem to be fitting more loosely, and several friends have asked if you have lost weight. If at this point your bathroom scale indicated that you had lost 10 pounds, this would make sense and you would continue to use the scale. But if it indicated that you had gained 10 pounds, you would rightly conclude that it was broken and either fix it or get rid of it. In evaluating a measurement method, psychologists consider two general dimensions: reliability and validity.

Reliability

Reliability refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (interrater reliability).

Test-Retest Reliability

When researchers measure a construct that they assume to be consistent across time, then the scores they obtain should also be consistent across time. Test-retest reliability is the extent to which this is actually the case. For example, intelligence is generally thought to be consistent across time. A person who is highly intelligent today will be highly intelligent next week. This means that any good measure of intelligence should produce roughly the same scores for this individual next week as it does today. Clearly, a measure that produces highly inconsistent scores over time cannot be a very good measure of a construct that is supposed to be consistent.

Assessing test-retest reliability requires using the measure on a group of people at one time, using it again on the same group of people at a later time, and then looking at test-retest correlation between the two sets of scores. This is typically done by graphing the data in a scatterplot and computing Pearson’s r . Figure 5.3 “Test-Retest Correlation Between Two Sets of Scores of Several College Students on the Rosenberg Self-Esteem Scale, Given Two Times a Week Apart” shows the correlation between two sets of scores of several college students on the Rosenberg Self-Esteem Scale, given two times a week apart. Pearson’s r for these data is +.95. In general, a test-retest correlation of +.80 or greater is considered to indicate good reliability.

Figure 5.3 Test-Retest Correlation Between Two Sets of Scores of Several College Students on the Rosenberg Self-Esteem Scale, Given Two Times a Week Apart

Again, high test-retest correlations make sense when the construct being measured is assumed to be consistent over time, which is the case for intelligence, self-esteem, and the Big Five personality dimensions. But other constructs are not assumed to be stable over time. The very nature of mood, for example, is that it changes. So a measure of mood that produced a low test-retest correlation over a period of a month would not be a cause for concern.

Internal Consistency

A second kind of reliability is internal consistency , which is the consistency of people’s responses across the items on a multiple-item measure. In general, all the items on such measures are supposed to reflect the same underlying construct, so people’s scores on those items should be correlated with each other. On the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to agree that that they have a number of good qualities. If people’s responses to the different items are not correlated with each other, then it would no longer make sense to claim that they are all measuring the same underlying construct. This is as true for behavioral and physiological measures as for self-report measures. For example, people might make a series of bets in a simulated game of roulette as a measure of their level of risk seeking. This measure would be internally consistent to the extent that individual participants’ bets were consistently high or low across trials.

Like test-retest reliability, internal consistency can only be assessed by collecting and analyzing data. One approach is to look at a split-half correlation . This involves splitting the items into two sets, such as the first and second halves of the items or the even- and odd-numbered items. Then a score is computed for each set of items, and the relationship between the two sets of scores is examined. For example, Figure 5.4 “Split-Half Correlation Between Several College Students’ Scores on the Even-Numbered Items and Their Scores on the Odd-Numbered Items of the Rosenberg Self-Esteem Scale” shows the split-half correlation between several college students’ scores on the even-numbered items and their scores on the odd-numbered items of the Rosenberg Self-Esteem Scale. Pearson’s r for these data is +.88. A split-half correlation of +.80 or greater is generally considered good internal consistency.

Figure 5.4 Split-Half Correlation Between Several College Students’ Scores on the Even-Numbered Items and Their Scores on the Odd-Numbered Items of the Rosenberg Self-Esteem Scale

Perhaps the most common measure of internal consistency used by researchers in psychology is a statistic called Cronbach’s α (the Greek letter alpha). Conceptually, α is the mean of all possible split-half correlations for a set of items. For example, there are 252 ways to split a set of 10 items into two sets of five. Cronbach’s α would be the mean of the 252 split-half correlations. Note that this is not how α is actually computed, but it is a correct way of interpreting the meaning of this statistic. Again, a value of +.80 or greater is generally taken to indicate good internal consistency.

Interrater Reliability

Many behavioral measures involve significant judgment on the part of an observer or a rater. Interrater reliability is the extent to which different observers are consistent in their judgments. For example, if you were interested in measuring college students’ social skills, you could make video recordings of them as they interacted with another student whom they are meeting for the first time. Then you could have two or more observers watch the videos and rate each student’s level of social skills. To the extent that each participant does in fact have some level of social skills that can be detected by an attentive observer, different observers’ ratings should be highly correlated with each other. If they were not, then those ratings could not be an accurate representation of participants’ social skills. Interrater reliability is often assessed using Cronbach’s α when the judgments are quantitative or an analogous statistic called Cohen’s κ (the Greek letter kappa) when they are categorical.

Validity is the extent to which the scores from a measure represent the variable they are intended to. But how do researchers make this judgment? We have already considered one factor that they take into account—reliability. When a measure has good test-retest reliability and internal consistency, researchers should be more confident that the scores represent what they are supposed to. There has to be more to it, however, because a measure can be extremely reliable but have no validity whatsoever. As an absurd example, imagine someone who believes that people’s index finger length reflects their self-esteem and therefore tries to measure self-esteem by holding a ruler up to people’s index fingers. Although this measure would have extremely good test-retest reliability, it would have absolutely no validity. The fact that one person’s index finger is a centimeter longer than another’s would indicate nothing about which one had higher self-esteem.

Textbook presentations of validity usually divide it into several distinct “types.” But a good way to interpret these types is that they are other kinds of evidence—in addition to reliability—that should be taken into account when judging the validity of a measure. Here we consider four basic kinds: face validity, content validity, criterion validity, and discriminant validity.

Face Validity

Face validity is the extent to which a measurement method appears “on its face” to measure the construct of interest. Most people would expect a self-esteem questionnaire to include items about whether they see themselves as a person of worth and whether they think they have good qualities. So a questionnaire that included these kinds of items would have good face validity. The finger-length method of measuring self-esteem, on the other hand, seems to have nothing to do with self-esteem and therefore has poor face validity. Although face validity can be assessed quantitatively—for example, by having a large sample of people rate a measure in terms of whether it appears to measure what it is intended to—it is usually assessed informally.

Face validity is at best a very weak kind of evidence that a measurement method is measuring what it is supposed to. One reason is that it is based on people’s intuitions about human behavior, which are frequently wrong. It is also the case that many established measures in psychology work quite well despite lacking face validity. The Minnesota Multiphasic Personality Inventory (MMPI) measures many personality characteristics and disorders by having people decide whether each of over 567 different statements applies to them—where many of the statements do not have any obvious relationship to the construct that they measure. Another example is the Implicit Association Test, which measures prejudice in a way that is nonintuitive to most people (see Note 5.31 “How Prejudiced Are You?” ).

How Prejudiced Are You?

The Implicit Association Test (IAT) is used to measure people’s attitudes toward various social groups. The IAT is a behavioral measure designed to reveal negative attitudes that people might not admit to on a self-report measure. It focuses on how quickly people are able to categorize words and images representing two contrasting groups (e.g., gay and straight) along with other positive and negative stimuli (e.g., the words “wonderful” or “nasty”). The IAT has been used in dozens of published research studies, and there is strong evidence for both its reliability and its validity (Nosek, Greenwald, & Banaji, 2006). You can learn more about the IAT—and take several of them for yourself—at the following website: https://implicit.harvard.edu/implicit .

Content Validity

Content validity is the extent to which a measure “covers” the construct of interest. For example, if a researcher conceptually defines test anxiety as involving both sympathetic nervous system activation (leading to nervous feelings) and negative thoughts, then his measure of test anxiety should include items about both nervous feelings and negative thoughts. Or consider that attitudes are usually defined as involving thoughts, feelings, and actions toward something. By this conceptual definition, a person has a positive attitude toward exercise to the extent that he or she thinks positive thoughts about exercising, feels good about exercising, and actually exercises. So to have good content validity, a measure of people’s attitudes toward exercise would have to reflect all three of these aspects. Like face validity, content validity is not usually assessed quantitatively. Instead, it is assessed by carefully checking the measurement method against the conceptual definition of the construct.

Criterion Validity

Criterion validity is the extent to which people’s scores on a measure are correlated with other variables (known as criteria ) that one would expect them to be correlated with. For example, people’s scores on a new measure of test anxiety should be negatively correlated with their performance on an important school exam. If it were found that people’s scores were in fact negatively correlated with their exam performance, then this would be a piece of evidence that these scores really represent people’s test anxiety. But if it were found that people scored equally well on the exam regardless of their test anxiety scores, then this would cast doubt on the validity of the measure.

A criterion can be any variable that one has reason to think should be correlated with the construct being measured, and there will usually be many of them. For example, one would expect test anxiety scores to be negatively correlated with exam performance and course grades and positively correlated with general anxiety and with blood pressure during an exam. Or imagine that a researcher develops a new measure of physical risk taking. People’s scores on this measure should be correlated with their participation in “extreme” activities such as snowboarding and rock climbing, the number of speeding tickets they have received, and even the number of broken bones they have had over the years. Criteria can also include other measures of the same construct. For example, one would expect new measures of test anxiety or physical risk taking to be positively correlated with existing measures of the same constructs. So the use of converging operations is one way to examine criterion validity.

Assessing criterion validity requires collecting data using the measure. Researchers John Cacioppo and Richard Petty did this when they created their self-report Need for Cognition Scale to measure how much people value and engage in thinking (Cacioppo & Petty, 1982). In a series of studies, they showed that college faculty scored higher than assembly-line workers, that people’s scores were positively correlated with their scores on a standardized academic achievement test, and that their scores were negatively correlated with their scores on a measure of dogmatism (which represents a tendency toward obedience). In the years since it was created, the Need for Cognition Scale has been used in literally hundreds of studies and has been shown to be correlated with a wide variety of other variables, including the effectiveness of an advertisement, interest in politics, and juror decisions (Petty, Briñol, Loersch, & McCaslin, 2009).

Discriminant Validity

Discriminant validity is the extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct. For example, self-esteem is a general attitude toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one happens to be feeling right now. So people’s scores on a new measure of self-esteem should not be very highly correlated with their moods. If the new measure of self-esteem were highly correlated with a measure of mood, it could be argued that the new measure is not really measuring self-esteem; it is measuring mood instead.

When they created the Need for Cognition Scale, Cacioppo and Petty also provided evidence of discriminant validity by showing that people’s scores were not correlated with certain other variables. For example, they found only a weak correlation between people’s need for cognition and a measure of their cognitive style—the extent to which they tend to think analytically by breaking ideas into smaller parts or holistically in terms of “the big picture.” They also found no correlation between people’s need for cognition and measures of their test anxiety and their tendency to respond in socially desirable ways. All these low correlations provide evidence that the measure is reflecting a conceptually distinct construct.

Key Takeaways

• Psychological researchers do not simply assume that their measures work. Instead, they conduct research to show that they work. If they cannot show that they work, they stop using them.
• There are two distinct criteria by which researchers evaluate their measures: reliability and validity. Reliability is consistency across time (test-retest reliability), across items (internal consistency), and across researchers (interrater reliability). Validity is the extent to which the scores actually represent the variable they are intended to.
• Validity is a judgment based on various types of evidence. The relevant evidence includes the measure’s reliability, whether it covers the construct of interest, and whether the scores it produces are correlated with other variables they are expected to be correlated with and not correlated with variables that are conceptually distinct.
• The reliability and validity of a measure is not established by any single study but by the pattern of results across multiple studies. The assessment of reliability and validity is an ongoing process.
• Practice: Ask several friends to complete the Rosenberg Self-Esteem Scale. Then assess its internal consistency by making a scatterplot to show the split-half correlation (even- vs. odd-numbered items). Compute Pearson’s r too if you know how.
• Discussion: Think back to the last college exam you took and think of the exam as a psychological measure. What construct do you think it was intended to measure? Comment on its face and content validity. What data could you collect to assess its reliability, criterion validity, and discriminant validity?
• Practice: Take an Implicit Association Test and then list as many ways to assess its criterion validity as you can think of.

Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42 , 116–131.

Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2006). The Implicit Association Test at age 7: A methodological and conceptual review. In J. A. Bargh (Ed.), Social psychology and the unconscious: The automaticity of higher mental processes (pp. 265–292). London, England: Psychology Press.

Petty, R. E, Briñol, P., Loersch, C., & McCaslin, M. J. (2009). The need for cognition. In M. R. Leary & R. H. Hoyle (Eds.), Handbook of individual differences in social behavior (pp. 318–329). New York, NY: Guilford Press.

Research Methods in Psychology Copyright © 2016 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• The 4 Types of Reliability in Research | Definitions & Examples

The 4 Types of Reliability in Research | Definitions & Examples

Published on 3 May 2022 by Fiona Middleton . Revised on 26 August 2022.

Reliability tells you how consistently a method measures something. When you apply the same method to the same   sample   under the same conditions, you should get the same results. If not, the method of measurement may be unreliable.

There are four main types of reliability. Each can be estimated by comparing different sets of results produced by the same method.

Table of contents

Test-retest reliability, interrater reliability, parallel forms reliability, internal consistency, which type of reliability applies to my research.

Test-retest reliability measures the consistency of results when you repeat the same test on the same sample at a different point in time. You use it when you are measuring something that you expect to stay constant in your sample.

Why test-retest reliability is important

Many factors can influence your results at different points in time: for example, respondents might experience different moods, or external conditions might affect their ability to respond accurately.

Test-retest reliability can be used to assess how well a method resists these factors over time. The smaller the difference between the two sets of results, the higher the test-retest reliability.

How to measure test-retest reliability

To measure test-retest reliability, you conduct the same test on the same group of people at two different points in time. Then you calculate the correlation between the two sets of results.

Improving test-retest reliability

• When designing tests or questionnaires , try to formulate questions, statements, and tasks in a way that won’t be influenced by the mood or concentration of participants.
• When planning your methods of data collection , try to minimise the influence of external factors, and make sure all samples are tested under the same conditions.
• Remember that changes can be expected to occur in the participants over time, and take these into account.

Prevent plagiarism, run a free check.

Inter-rater reliability (also called inter-observer reliability) measures the degree of agreement between different people observing or assessing the same thing. You use it when data is collected by researchers assigning ratings, scores or categories to one or more variables .

Why inter-rater reliability is important

People are subjective, so different observers’ perceptions of situations and phenomena naturally differ. Reliable research aims to minimise subjectivity as much as possible so that a different researcher could replicate the same results.

When designing the scale and criteria for data collection, it’s important to make sure that different people will rate the same variable consistently with minimal bias. This is especially important when there are multiple researchers involved in data collection or analysis.

How to measure inter-rater reliability

To measure inter-rater reliability, different researchers conduct the same measurement or observation on the same sample. Then you calculate the correlation between their different sets of results. If all the researchers give similar ratings, the test has high inter-rater reliability.

Improving inter-rater reliability

• Clearly define your variables and the methods that will be used to measure them.
• Develop detailed, objective criteria for how the variables will be rated, counted, or categorised.
• If multiple researchers are involved, ensure that they all have exactly the same information and training.

Parallel forms reliability measures the correlation between two equivalent versions of a test. You use it when you have two different assessment tools or sets of questions designed to measure the same thing.

Why parallel forms reliability is important

If you want to use multiple different versions of a test (for example, to avoid respondents repeating the same answers from memory), you first need to make sure that all the sets of questions or measurements give reliable results.

How to measure parallel forms reliability

The most common way to measure parallel forms reliability is to produce a large set of questions to evaluate the same thing, then divide these randomly into two question sets.

The same group of respondents answers both sets, and you calculate the correlation between the results. High correlation between the two indicates high parallel forms reliability.

Improving parallel forms reliability

• Ensure that all questions or test items are based on the same theory and formulated to measure the same thing.

Internal consistency assesses the correlation between multiple items in a test that are intended to measure the same construct.

You can calculate internal consistency without repeating the test or involving other researchers, so it’s a good way of assessing reliability when you only have one dataset.

Why internal consistency is important

When you devise a set of questions or ratings that will be combined into an overall score, you have to make sure that all of the items really do reflect the same thing. If responses to different items contradict one another, the test might be unreliable.

How to measure internal consistency

Two common methods are used to measure internal consistency.

• Average inter-item correlation : For a set of measures designed to assess the same construct, you calculate the correlation between the results of all possible pairs of items and then calculate the average.
• Split-half reliability : You randomly split a set of measures into two sets. After testing the entire set on the respondents, you calculate the correlation between the two sets of responses.

Improving internal consistency

• Take care when devising questions or measures: those intended to reflect the same concept should be based on the same theory and carefully formulated.

It’s important to consider reliability when planning your research design , collecting and analysing your data, and writing up your research. The type of reliability you should calculate depends on the type of research  and your  methodology .

If possible and relevant, you should statistically calculate reliability and state this alongside your results .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Middleton, F. (2022, August 26). The 4 Types of Reliability in Research | Definitions & Examples. Scribbr. Retrieved 21 May 2024, from https://www.scribbr.co.uk/research-methods/reliability-explained/

Is this article helpful?

Fiona Middleton

Other students also liked, reliability vs validity in research | differences, types & examples, the 4 types of validity | types, definitions & examples, a quick guide to experimental design | 5 steps & examples.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 5: Psychological Measurement

Reliability and Validity of Measurement

Learning Objectives

• Define reliability, including the different types and how they are assessed.
• Define validity, including the different types and how they are assessed.
• Describe the kinds of evidence that would be relevant to assessing the reliability and validity of a particular measure.

Again, measurement involves assigning scores to individuals so that they represent some characteristic of the individuals. But how do researchers know that the scores actually represent the characteristic, especially when it is a construct like intelligence, self-esteem, depression, or working memory capacity? The answer is that they conduct research using the measure to confirm that the scores make sense based on their understanding of the construct being measured. This is an extremely important point. Psychologists do not simply  assume  that their measures work. Instead, they collect data to demonstrate  that they work. If their research does not demonstrate that a measure works, they stop using it.

As an informal example, imagine that you have been dieting for a month. Your clothes seem to be fitting more loosely, and several friends have asked if you have lost weight. If at this point your bathroom scale indicated that you had lost 10 pounds, this would make sense and you would continue to use the scale. But if it indicated that you had gained 10 pounds, you would rightly conclude that it was broken and either fix it or get rid of it. In evaluating a measurement method, psychologists consider two general dimensions: reliability and validity.

Reliability

Reliability  refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (inter-rater reliability).

Test-Retest Reliability

When researchers measure a construct that they assume to be consistent across time, then the scores they obtain should also be consistent across time.  Test-retest reliability  is the extent to which this is actually the case. For example, intelligence is generally thought to be consistent across time. A person who is highly intelligent today will be highly intelligent next week. This means that any good measure of intelligence should produce roughly the same scores for this individual next week as it does today. Clearly, a measure that produces highly inconsistent scores over time cannot be a very good measure of a construct that is supposed to be consistent.

Assessing test-retest reliability requires using the measure on a group of people at one time, using it again on the  same  group of people at a later time, and then looking at  test-retest correlation  between the two sets of scores. This is typically done by graphing the data in a scatterplot and computing Pearson’s  r . Figure 5.2 shows the correlation between two sets of scores of several university students on the Rosenberg Self-Esteem Scale, administered two times, a week apart. Pearson’s r for these data is +.95. In general, a test-retest correlation of +.80 or greater is considered to indicate good reliability.

Again, high test-retest correlations make sense when the construct being measured is assumed to be consistent over time, which is the case for intelligence, self-esteem, and the Big Five personality dimensions. But other constructs are not assumed to be stable over time. The very nature of mood, for example, is that it changes. So a measure of mood that produced a low test-retest correlation over a period of a month would not be a cause for concern.

Internal Consistency

A second kind of reliability is  internal consistency , which is the consistency of people’s responses across the items on a multiple-item measure. In general, all the items on such measures are supposed to reflect the same underlying construct, so people’s scores on those items should be correlated with each other. On the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to agree that that they have a number of good qualities. If people’s responses to the different items are not correlated with each other, then it would no longer make sense to claim that they are all measuring the same underlying construct. This is as true for behavioural and physiological measures as for self-report measures. For example, people might make a series of bets in a simulated game of roulette as a measure of their level of risk seeking. This measure would be internally consistent to the extent that individual participants’ bets were consistently high or low across trials.

Like test-retest reliability, internal consistency can only be assessed by collecting and analyzing data. One approach is to look at a  split-half correlation . This involves splitting the items into two sets, such as the first and second halves of the items or the even- and odd-numbered items. Then a score is computed for each set of items, and the relationship between the two sets of scores is examined. For example, Figure 5.3 shows the split-half correlation between several university students’ scores on the even-numbered items and their scores on the odd-numbered items of the Rosenberg Self-Esteem Scale. Pearson’s  r  for these data is +.88. A split-half correlation of +.80 or greater is generally considered good internal consistency.

Perhaps the most common measure of internal consistency used by researchers in psychology is a statistic called  Cronbach’s α  (the Greek letter alpha). Conceptually, α is the mean of all possible split-half correlations for a set of items. For example, there are 252 ways to split a set of 10 items into two sets of five. Cronbach’s α would be the mean of the 252 split-half correlations. Note that this is not how α is actually computed, but it is a correct way of interpreting the meaning of this statistic. Again, a value of +.80 or greater is generally taken to indicate good internal consistency.

Interrater Reliability

Many behavioural measures involve significant judgment on the part of an observer or a rater.  Inter-rater reliability  is the extent to which different observers are consistent in their judgments. For example, if you were interested in measuring university students’ social skills, you could make video recordings of them as they interacted with another student whom they are meeting for the first time. Then you could have two or more observers watch the videos and rate each student’s level of social skills. To the extent that each participant does in fact have some level of social skills that can be detected by an attentive observer, different observers’ ratings should be highly correlated with each other. Inter-rater reliability would also have been measured in Bandura’s Bobo doll study. In this case, the observers’ ratings of how many acts of aggression a particular child committed while playing with the Bobo doll should have been highly positively correlated. Interrater reliability is often assessed using Cronbach’s α when the judgments are quantitative or an analogous statistic called Cohen’s κ (the Greek letter kappa) when they are categorical.

Validity  is the extent to which the scores from a measure represent the variable they are intended to. But how do researchers make this judgment? We have already considered one factor that they take into account—reliability. When a measure has good test-retest reliability and internal consistency, researchers should be more confident that the scores represent what they are supposed to. There has to be more to it, however, because a measure can be extremely reliable but have no validity whatsoever. As an absurd example, imagine someone who believes that people’s index finger length reflects their self-esteem and therefore tries to measure self-esteem by holding a ruler up to people’s index fingers. Although this measure would have extremely good test-retest reliability, it would have absolutely no validity. The fact that one person’s index finger is a centimetre longer than another’s would indicate nothing about which one had higher self-esteem.

Discussions of validity usually divide it into several distinct “types.” But a good way to interpret these types is that they are other kinds of evidence—in addition to reliability—that should be taken into account when judging the validity of a measure. Here we consider three basic kinds: face validity, content validity, and criterion validity.

Face Validity

Face validity  is the extent to which a measurement method appears “on its face” to measure the construct of interest. Most people would expect a self-esteem questionnaire to include items about whether they see themselves as a person of worth and whether they think they have good qualities. So a questionnaire that included these kinds of items would have good face validity. The finger-length method of measuring self-esteem, on the other hand, seems to have nothing to do with self-esteem and therefore has poor face validity. Although face validity can be assessed quantitatively—for example, by having a large sample of people rate a measure in terms of whether it appears to measure what it is intended to—it is usually assessed informally.

Face validity is at best a very weak kind of evidence that a measurement method is measuring what it is supposed to. One reason is that it is based on people’s intuitions about human behaviour, which are frequently wrong. It is also the case that many established measures in psychology work quite well despite lacking face validity. The Minnesota Multiphasic Personality Inventory-2 (MMPI-2) measures many personality characteristics and disorders by having people decide whether each of over 567 different statements applies to them—where many of the statements do not have any obvious relationship to the construct that they measure. For example, the items “I enjoy detective or mystery stories” and “The sight of blood doesn’t frighten me or make me sick” both measure the suppression of aggression. In this case, it is not the participants’ literal answers to these questions that are of interest, but rather whether the pattern of the participants’ responses to a series of questions matches those of individuals who tend to suppress their aggression.

Content Validity

Content validity  is the extent to which a measure “covers” the construct of interest. For example, if a researcher conceptually defines test anxiety as involving both sympathetic nervous system activation (leading to nervous feelings) and negative thoughts, then his measure of test anxiety should include items about both nervous feelings and negative thoughts. Or consider that attitudes are usually defined as involving thoughts, feelings, and actions toward something. By this conceptual definition, a person has a positive attitude toward exercise to the extent that he or she thinks positive thoughts about exercising, feels good about exercising, and actually exercises. So to have good content validity, a measure of people’s attitudes toward exercise would have to reflect all three of these aspects. Like face validity, content validity is not usually assessed quantitatively. Instead, it is assessed by carefully checking the measurement method against the conceptual definition of the construct.

Criterion Validity

Criterion validity  is the extent to which people’s scores on a measure are correlated with other variables (known as  criteria ) that one would expect them to be correlated with. For example, people’s scores on a new measure of test anxiety should be negatively correlated with their performance on an important school exam. If it were found that people’s scores were in fact negatively correlated with their exam performance, then this would be a piece of evidence that these scores really represent people’s test anxiety. But if it were found that people scored equally well on the exam regardless of their test anxiety scores, then this would cast doubt on the validity of the measure.

A criterion can be any variable that one has reason to think should be correlated with the construct being measured, and there will usually be many of them. For example, one would expect test anxiety scores to be negatively correlated with exam performance and course grades and positively correlated with general anxiety and with blood pressure during an exam. Or imagine that a researcher develops a new measure of physical risk taking. People’s scores on this measure should be correlated with their participation in “extreme” activities such as snowboarding and rock climbing, the number of speeding tickets they have received, and even the number of broken bones they have had over the years. When the criterion is measured at the same time as the construct, criterion validity is referred to as concurrent validity ; however, when the criterion is measured at some point in the future (after the construct has been measured), it is referred to as predictive validity (because scores on the measure have “predicted” a future outcome).

Criteria can also include other measures of the same construct. For example, one would expect new measures of test anxiety or physical risk taking to be positively correlated with existing measures of the same constructs. This is known as convergent validity .

Assessing convergent validity requires collecting data using the measure. Researchers John Cacioppo and Richard Petty did this when they created their self-report Need for Cognition Scale to measure how much people value and engage in thinking (Cacioppo & Petty, 1982) [1] . In a series of studies, they showed that people’s scores were positively correlated with their scores on a standardized academic achievement test, and that their scores were negatively correlated with their scores on a measure of dogmatism (which represents a tendency toward obedience). In the years since it was created, the Need for Cognition Scale has been used in literally hundreds of studies and has been shown to be correlated with a wide variety of other variables, including the effectiveness of an advertisement, interest in politics, and juror decisions (Petty, Briñol, Loersch, & McCaslin, 2009) [2] .

Discriminant Validity

Discriminant validity , on the other hand, is the extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct. For example, self-esteem is a general attitude toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one happens to be feeling right now. So people’s scores on a new measure of self-esteem should not be very highly correlated with their moods. If the new measure of self-esteem were highly correlated with a measure of mood, it could be argued that the new measure is not really measuring self-esteem; it is measuring mood instead.

When they created the Need for Cognition Scale, Cacioppo and Petty also provided evidence of discriminant validity by showing that people’s scores were not correlated with certain other variables. For example, they found only a weak correlation between people’s need for cognition and a measure of their cognitive style—the extent to which they tend to think analytically by breaking ideas into smaller parts or holistically in terms of “the big picture.” They also found no correlation between people’s need for cognition and measures of their test anxiety and their tendency to respond in socially desirable ways. All these low correlations provide evidence that the measure is reflecting a conceptually distinct construct.

Key Takeaways

• Psychological researchers do not simply assume that their measures work. Instead, they conduct research to show that they work. If they cannot show that they work, they stop using them.
• There are two distinct criteria by which researchers evaluate their measures: reliability and validity. Reliability is consistency across time (test-retest reliability), across items (internal consistency), and across researchers (interrater reliability). Validity is the extent to which the scores actually represent the variable they are intended to.
• Validity is a judgment based on various types of evidence. The relevant evidence includes the measure’s reliability, whether it covers the construct of interest, and whether the scores it produces are correlated with other variables they are expected to be correlated with and not correlated with variables that are conceptually distinct.
• The reliability and validity of a measure is not established by any single study but by the pattern of results across multiple studies. The assessment of reliability and validity is an ongoing process.
• Practice: Ask several friends to complete the Rosenberg Self-Esteem Scale. Then assess its internal consistency by making a scatterplot to show the split-half correlation (even- vs. odd-numbered items). Compute Pearson’s  r too if you know how.
• Discussion: Think back to the last college exam you took and think of the exam as a psychological measure. What construct do you think it was intended to measure? Comment on its face and content validity. What data could you collect to assess its reliability and criterion validity?
• Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42 , 116–131. ↵
• Petty, R. E, Briñol, P., Loersch, C., & McCaslin, M. J. (2009). The need for cognition. In M. R. Leary & R. H. Hoyle (Eds.), Handbook of individual differences in social behaviour (pp. 318–329). New York, NY: Guilford Press. ↵

The consistency of a measure.

The consistency of a measure over time.

The consistency of a measure on the same group of people at different times.

Consistency of people’s responses across the items on a multiple-item measure.

Method of assessing internal consistency through splitting the items into two sets and examining the relationship between them.

A statistic in which α is the mean of all possible split-half correlations for a set of items.

The extent to which different observers are consistent in their judgments.

The extent to which the scores from a measure represent the variable they are intended to.

The extent to which a measurement method appears to measure the construct of interest.

The extent to which a measure “covers” the construct of interest.

The extent to which people’s scores on a measure are correlated with other variables that one would expect them to be correlated with.

In reference to criterion validity, variables that one would expect to be correlated with the measure.

When the criterion is measured at the same time as the construct.

when the criterion is measured at some point in the future (after the construct has been measured).

When new measures positively correlate with existing measures of the same constructs.

The extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct.

Research Methods in Psychology - 2nd Canadian Edition Copyright © 2015 by Paul C. Price, Rajiv Jhangiani, & I-Chant A. Chiang is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Family Med Prim Care
• v.4(3); Jul-Sep 2015

Validity, reliability, and generalizability in qualitative research

Lawrence leung.

1 Department of Family Medicine, Queen's University, Kingston, Ontario, Canada

2 Centre of Studies in Primary Care, Queen's University, Kingston, Ontario, Canada

In general practice, qualitative research contributes as significantly as quantitative research, in particular regarding psycho-social aspects of patient-care, health services provision, policy setting, and health administrations. In contrast to quantitative research, qualitative research as a whole has been constantly critiqued, if not disparaged, by the lack of consensus for assessing its quality and robustness. This article illustrates with five published studies how qualitative research can impact and reshape the discipline of primary care, spiraling out from clinic-based health screening to community-based disease monitoring, evaluation of out-of-hours triage services to provincial psychiatric care pathways model and finally, national legislation of core measures for children's healthcare insurance. Fundamental concepts of validity, reliability, and generalizability as applicable to qualitative research are then addressed with an update on the current views and controversies.

Nature of Qualitative Research versus Quantitative Research

The essence of qualitative research is to make sense of and recognize patterns among words in order to build up a meaningful picture without compromising its richness and dimensionality. Like quantitative research, the qualitative research aims to seek answers for questions of “how, where, when who and why” with a perspective to build a theory or refute an existing theory. Unlike quantitative research which deals primarily with numerical data and their statistical interpretations under a reductionist, logical and strictly objective paradigm, qualitative research handles nonnumerical information and their phenomenological interpretation, which inextricably tie in with human senses and subjectivity. While human emotions and perspectives from both subjects and researchers are considered undesirable biases confounding results in quantitative research, the same elements are considered essential and inevitable, if not treasurable, in qualitative research as they invariable add extra dimensions and colors to enrich the corpus of findings. However, the issue of subjectivity and contextual ramifications has fueled incessant controversies regarding yardsticks for quality and trustworthiness of qualitative research results for healthcare.

Impact of Qualitative Research upon Primary Care

In many ways, qualitative research contributes significantly, if not more so than quantitative research, to the field of primary care at various levels. Five qualitative studies are chosen to illustrate how various methodologies of qualitative research helped in advancing primary healthcare, from novel monitoring of chronic obstructive pulmonary disease (COPD) via mobile-health technology,[ 1 ] informed decision for colorectal cancer screening,[ 2 ] triaging out-of-hours GP services,[ 3 ] evaluating care pathways for community psychiatry[ 4 ] and finally prioritization of healthcare initiatives for legislation purposes at national levels.[ 5 ] With the recent advances of information technology and mobile connecting device, self-monitoring and management of chronic diseases via tele-health technology may seem beneficial to both the patient and healthcare provider. Recruiting COPD patients who were given tele-health devices that monitored lung functions, Williams et al. [ 1 ] conducted phone interviews and analyzed their transcripts via a grounded theory approach, identified themes which enabled them to conclude that such mobile-health setup and application helped to engage patients with better adherence to treatment and overall improvement in mood. Such positive findings were in contrast to previous studies, which opined that elderly patients were often challenged by operating computer tablets,[ 6 ] or, conversing with the tele-health software.[ 7 ] To explore the content of recommendations for colorectal cancer screening given out by family physicians, Wackerbarth, et al. [ 2 ] conducted semi-structure interviews with subsequent content analysis and found that most physicians delivered information to enrich patient knowledge with little regard to patients’ true understanding, ideas, and preferences in the matter. These findings suggested room for improvement for family physicians to better engage their patients in recommending preventative care. Faced with various models of out-of-hours triage services for GP consultations, Egbunike et al. [ 3 ] conducted thematic analysis on semi-structured telephone interviews with patients and doctors in various urban, rural and mixed settings. They found that the efficiency of triage services remained a prime concern from both users and providers, among issues of access to doctors and unfulfilled/mismatched expectations from users, which could arouse dissatisfaction and legal implications. In UK, a care pathways model for community psychiatry had been introduced but its benefits were unclear. Khandaker et al. [ 4 ] hence conducted a qualitative study using semi-structure interviews with medical staff and other stakeholders; adopting a grounded-theory approach, major themes emerged which included improved equality of access, more focused logistics, increased work throughput and better accountability for community psychiatry provided under the care pathway model. Finally, at the US national level, Mangione-Smith et al. [ 5 ] employed a modified Delphi method to gather consensus from a panel of nominators which were recognized experts and stakeholders in their disciplines, and identified a core set of quality measures for children's healthcare under the Medicaid and Children's Health Insurance Program. These core measures were made transparent for public opinion and later passed on for full legislation, hence illustrating the impact of qualitative research upon social welfare and policy improvement.

Overall Criteria for Quality in Qualitative Research

Given the diverse genera and forms of qualitative research, there is no consensus for assessing any piece of qualitative research work. Various approaches have been suggested, the two leading schools of thoughts being the school of Dixon-Woods et al. [ 8 ] which emphasizes on methodology, and that of Lincoln et al. [ 9 ] which stresses the rigor of interpretation of results. By identifying commonalities of qualitative research, Dixon-Woods produced a checklist of questions for assessing clarity and appropriateness of the research question; the description and appropriateness for sampling, data collection and data analysis; levels of support and evidence for claims; coherence between data, interpretation and conclusions, and finally level of contribution of the paper. These criteria foster the 10 questions for the Critical Appraisal Skills Program checklist for qualitative studies.[ 10 ] However, these methodology-weighted criteria may not do justice to qualitative studies that differ in epistemological and philosophical paradigms,[ 11 , 12 ] one classic example will be positivistic versus interpretivistic.[ 13 ] Equally, without a robust methodological layout, rigorous interpretation of results advocated by Lincoln et al. [ 9 ] will not be good either. Meyrick[ 14 ] argued from a different angle and proposed fulfillment of the dual core criteria of “transparency” and “systematicity” for good quality qualitative research. In brief, every step of the research logistics (from theory formation, design of study, sampling, data acquisition and analysis to results and conclusions) has to be validated if it is transparent or systematic enough. In this manner, both the research process and results can be assured of high rigor and robustness.[ 14 ] Finally, Kitto et al. [ 15 ] epitomized six criteria for assessing overall quality of qualitative research: (i) Clarification and justification, (ii) procedural rigor, (iii) sample representativeness, (iv) interpretative rigor, (v) reflexive and evaluative rigor and (vi) transferability/generalizability, which also double as evaluative landmarks for manuscript review to the Medical Journal of Australia. Same for quantitative research, quality for qualitative research can be assessed in terms of validity, reliability, and generalizability.

Validity in qualitative research means “appropriateness” of the tools, processes, and data. Whether the research question is valid for the desired outcome, the choice of methodology is appropriate for answering the research question, the design is valid for the methodology, the sampling and data analysis is appropriate, and finally the results and conclusions are valid for the sample and context. In assessing validity of qualitative research, the challenge can start from the ontology and epistemology of the issue being studied, e.g. the concept of “individual” is seen differently between humanistic and positive psychologists due to differing philosophical perspectives:[ 16 ] Where humanistic psychologists believe “individual” is a product of existential awareness and social interaction, positive psychologists think the “individual” exists side-by-side with formation of any human being. Set off in different pathways, qualitative research regarding the individual's wellbeing will be concluded with varying validity. Choice of methodology must enable detection of findings/phenomena in the appropriate context for it to be valid, with due regard to culturally and contextually variable. For sampling, procedures and methods must be appropriate for the research paradigm and be distinctive between systematic,[ 17 ] purposeful[ 18 ] or theoretical (adaptive) sampling[ 19 , 20 ] where the systematic sampling has no a priori theory, purposeful sampling often has a certain aim or framework and theoretical sampling is molded by the ongoing process of data collection and theory in evolution. For data extraction and analysis, several methods were adopted to enhance validity, including 1 st tier triangulation (of researchers) and 2 nd tier triangulation (of resources and theories),[ 17 , 21 ] well-documented audit trail of materials and processes,[ 22 , 23 , 24 ] multidimensional analysis as concept- or case-orientated[ 25 , 26 ] and respondent verification.[ 21 , 27 ]

Reliability

In quantitative research, reliability refers to exact replicability of the processes and the results. In qualitative research with diverse paradigms, such definition of reliability is challenging and epistemologically counter-intuitive. Hence, the essence of reliability for qualitative research lies with consistency.[ 24 , 28 ] A margin of variability for results is tolerated in qualitative research provided the methodology and epistemological logistics consistently yield data that are ontologically similar but may differ in richness and ambience within similar dimensions. Silverman[ 29 ] proposed five approaches in enhancing the reliability of process and results: Refutational analysis, constant data comparison, comprehensive data use, inclusive of the deviant case and use of tables. As data were extracted from the original sources, researchers must verify their accuracy in terms of form and context with constant comparison,[ 27 ] either alone or with peers (a form of triangulation).[ 30 ] The scope and analysis of data included should be as comprehensive and inclusive with reference to quantitative aspects if possible.[ 30 ] Adopting the Popperian dictum of falsifiability as essence of truth and science, attempted to refute the qualitative data and analytes should be performed to assess reliability.[ 31 ]

Generalizability

Most qualitative research studies, if not all, are meant to study a specific issue or phenomenon in a certain population or ethnic group, of a focused locality in a particular context, hence generalizability of qualitative research findings is usually not an expected attribute. However, with rising trend of knowledge synthesis from qualitative research via meta-synthesis, meta-narrative or meta-ethnography, evaluation of generalizability becomes pertinent. A pragmatic approach to assessing generalizability for qualitative studies is to adopt same criteria for validity: That is, use of systematic sampling, triangulation and constant comparison, proper audit and documentation, and multi-dimensional theory.[ 17 ] However, some researchers espouse the approach of analytical generalization[ 32 ] where one judges the extent to which the findings in one study can be generalized to another under similar theoretical, and the proximal similarity model, where generalizability of one study to another is judged by similarities between the time, place, people and other social contexts.[ 33 ] Thus said, Zimmer[ 34 ] questioned the suitability of meta-synthesis in view of the basic tenets of grounded theory,[ 35 ] phenomenology[ 36 ] and ethnography.[ 37 ] He concluded that any valid meta-synthesis must retain the other two goals of theory development and higher-level abstraction while in search of generalizability, and must be executed as a third level interpretation using Gadamer's concepts of the hermeneutic circle,[ 38 , 39 ] dialogic process[ 38 ] and fusion of horizons.[ 39 ] Finally, Toye et al. [ 40 ] reported the practicality of using “conceptual clarity” and “interpretative rigor” as intuitive criteria for assessing quality in meta-ethnography, which somehow echoed Rolfe's controversial aesthetic theory of research reports.[ 41 ]

Food for Thought

Despite various measures to enhance or ensure quality of qualitative studies, some researchers opined from a purist ontological and epistemological angle that qualitative research is not a unified, but ipso facto diverse field,[ 8 ] hence any attempt to synthesize or appraise different studies under one system is impossible and conceptually wrong. Barbour argued from a philosophical angle that these special measures or “technical fixes” (like purposive sampling, multiple-coding, triangulation, and respondent validation) can never confer the rigor as conceived.[ 11 ] In extremis, Rolfe et al. opined from the field of nursing research, that any set of formal criteria used to judge the quality of qualitative research are futile and without validity, and suggested that any qualitative report should be judged by the form it is written (aesthetic) and not by the contents (epistemic).[ 41 ] Rolfe's novel view is rebutted by Porter,[ 42 ] who argued via logical premises that two of Rolfe's fundamental statements were flawed: (i) “The content of research report is determined by their forms” may not be a fact, and (ii) that research appraisal being “subject to individual judgment based on insight and experience” will mean those without sufficient experience of performing research will be unable to judge adequately – hence an elitist's principle. From a realism standpoint, Porter then proposes multiple and open approaches for validity in qualitative research that incorporate parallel perspectives[ 43 , 44 ] and diversification of meanings.[ 44 ] Any work of qualitative research, when read by the readers, is always a two-way interactive process, such that validity and quality has to be judged by the receiving end too and not by the researcher end alone.

In summary, the three gold criteria of validity, reliability and generalizability apply in principle to assess quality for both quantitative and qualitative research, what differs will be the nature and type of processes that ontologically and epistemologically distinguish between the two.

Source of Support: Nil.

Conflict of Interest: None declared.

Reliability In Psychology Research: Definitions & Examples

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

On This Page:

Reliability in psychology research refers to the reproducibility or consistency of measurements. Specifically, it is the degree to which a measurement instrument or procedure yields the same results on repeated trials. A measure is considered reliable if it produces consistent scores across different instances when the underlying thing being measured has not changed.

Reliability ensures that responses are consistent across times and occasions for instruments like questionnaires . Multiple forms of reliability exist, including test-retest, inter-rater, and internal consistency.

For example, if people weigh themselves during the day, they would expect to see a similar reading. Scales that measured weight differently each time would be of little use.

The same analogy could be applied to a tape measure that measures inches differently each time it is used. It would not be considered reliable.

If findings from research are replicated consistently, they are reliable. A correlation coefficient can be used to assess the degree of reliability. If a test is reliable, it should show a high positive correlation.

Of course, it is unlikely the same results will be obtained each time as participants and situations vary. Still, a strong positive correlation between the same test results indicates reliability.

Reliability is important because unreliable measures introduce random error that attenuates correlations and makes it harder to detect real relationships.

Ensuring high reliability for key measures in psychology research helps boost the sensitivity, validity, and replicability of studies. Estimating and reporting reliable evidence is considered an important methodological practice.

There are two types of reliability: internal and external.

• Internal reliability refers to how consistently different items within a single test measure the same concept or construct. It ensures that a test is stable across its components.
• External reliability measures how consistently a test produces similar results over repeated administrations or under different conditions. It ensures that a test is stable over time and situations.

Some key aspects of reliability in psychology research include:

• Test-retest reliability : The consistency of scores for the same person across two or more separate administrations of the same measurement procedure over time. High test-retest reliability suggests the measure provides a stable, reproducible score.
• Interrater reliability : The level of agreement in scores on a measure between different raters or observers rating the same target. High interrater reliability suggests the ratings are objective and not overly influenced by rater subjectivity or bias.
• Internal consistency reliability : The degree to which different test items or parts of an instrument that measure the same construct yield similar results. Analyzed statistically using Cronbach’s alpha, a high value suggests the items measure the same underlying concept.

Test-Retest Reliability

The test-retest method assesses the external consistency of a test. Examples of appropriate tests include questionnaires and psychometric tests. It measures the stability of a test over time.

A typical assessment would involve giving participants the same test on two separate occasions. If the same or similar results are obtained, then external reliability is established.

Here’s how it works:

• A test or measurement is administered to participants at one point in time.
• After a certain period, the same test is administered again to the same participants without any intervention or treatment in between.
• The scores from the two administrations are then correlated using a statistical method, often Pearson’s correlation.
• A high correlation between the scores from the two test administrations indicates good test-retest reliability, suggesting the test yields consistent results over time.

This method is especially useful for tests that measure stable traits or characteristics that aren’t expected to change over short periods.

The disadvantage of the test-retest method is that it takes a long time for results to be obtained. The reliability can be influenced by the time interval between tests and any events that might affect participants’ responses during this interval.

Beck et al. (1996) studied the responses of 26 outpatients on two separate therapy sessions one week apart, they found a correlation of .93 therefore demonstrating high test-restest reliability of the depression inventory.

This is an example of why reliability in psychological research is necessary, if it wasn’t for the reliability of such tests some individuals may not be successfully diagnosed with disorders such as depression and consequently will not be given appropriate therapy.

The timing of the test is important; if the duration is too brief, then participants may recall information from the first test, which could bias the results.

Alternatively, if the duration is too long, it is feasible that the participants could have changed in some important way which could also bias the results.

The test-retest method assesses the external consistency of a test. This refers to the degree to which different raters give consistent estimates of the same behavior. Inter-rater reliability can be used for interviews.

Inter-Rater Reliability

Inter-rater reliability, often termed inter-observer reliability, refers to the extent to which different raters or evaluators agree in assessing a particular phenomenon, behavior, or characteristic. It’s a measure of consistency and agreement between individuals scoring or evaluating the same items or behaviors.

High inter-rater reliability indicates that the findings or measurements are consistent across different raters, suggesting the results are not due to random chance or subjective biases of individual raters.

Statistical measures, such as Cohen’s Kappa or the Intraclass Correlation Coefficient (ICC), are often employed to quantify the level of agreement between raters, helping to ensure that findings are objective and reproducible.

Ensuring high inter-rater reliability is essential, especially in studies involving subjective judgment or observations, as it provides confidence that the findings are replicable and not heavily influenced by individual rater biases.

Note it can also be called inter-observer reliability when referring to observational research. Here, researchers observe the same behavior independently (to avoid bias) and compare their data. If the data is similar, then it is reliable.

Where observer scores do not significantly correlate, then reliability can be improved by:

• Train observers in the observation techniques and ensure everyone agrees with them.
• Ensuring behavior categories have been operationalized. This means that they have been objectively defined.

For example, if two researchers are observing ‘aggressive behavior’ of children at nursery they would both have their own subjective opinion regarding what aggression comprises.

In this scenario, they would be unlikely to record aggressive behavior the same, and the data would be unreliable.

However, if they were to operationalize the behavior category of aggression, this would be more objective and make it easier to identify when a specific behavior occurs.

For example, while “aggressive behavior” is subjective and not operationalized, “pushing” is objective and operationalized. Thus, researchers could count how many times children push each other over a certain duration of time.

Internal Consistency Reliability

Internal consistency reliability refers to how well different items on a test or survey that are intended to measure the same construct produce similar scores.

For example, a questionnaire measuring depression may have multiple questions tapping issues like sadness, changes in sleep and appetite, fatigue, and loss of interest. The assumption is that people’s responses across these different symptom items should be fairly consistent.

Cronbach’s alpha is a common statistic used to quantify internal consistency reliability. It calculates the average inter-item correlations among the test items. Values range from 0 to 1, with higher values indicating greater internal consistency. A good rule of thumb is that alpha should generally be above .70 to suggest adequate reliability.

An alpha of .90 for a depression questionnaire, for example, means there is a high average correlation between respondents’ scores on the different symptom items.

This suggests all the items are measuring the same underlying construct (depression) in a consistent manner. It taps the unidimensionality of the scale – evidence it is measuring one thing.

If some items were unrelated to others, the average inter-item correlations would be lower, resulting in a lower alpha. This would indicate the presence of multiple dimensions in the scale, rather than a unified single concept.

So, in summary, high internal consistency reliability evidenced through high Cronbach’s alpha provides support for the fact that various test items successfully tap into the same latent variable the researcher intends to measure. It suggests the items meaningfully cohere together to reliably measure that construct.

Split-Half Method

The split-half method assesses the internal consistency of a test, such as psychometric tests and questionnaires.

There, it measures the extent to which all parts of the test contribute equally to what is being measured.

The split-half approach provides another method of quantifying internal consistency by taking advantage of the natural variation when a single test is divided in half.

It’s somewhat cumbersome to implement but avoids limitations associated with Cronbach’s alpha. However, alpha remains much more widely used in practice due to its relative ease of calculation.

• A test or questionnaire is split into two halves, typically by separating even-numbered items from odd-numbered items, or first-half items vs. second-half.
• Each half is scored separately, and the scores are correlated using a statistical method, often Pearson’s correlation.
• The correlation between the two halves gives an indication of the test’s reliability. A higher correlation suggests better reliability.
• To adjust for the test’s shortened length (because we’ve split it in half), the Spearman-Brown prophecy formula is often applied to estimate the reliability of the full test based on the split-half reliability.

The reliability of a test could be improved by using this method. For example, any items on separate halves of a test with a low correlation (e.g., r = .25) should either be removed or rewritten.

The split-half method is a quick and easy way to establish reliability. However, it can only be effective with large questionnaires in which all questions measure the same construct. This means it would not be appropriate for tests that measure different constructs.

For example, the Minnesota Multiphasic Personality Inventory has sub scales measuring differently behaviors such as depression, schizophrenia, social introversion. Therefore the split-half method was not be an appropriate method to assess reliability for this personality test.

Validity vs. Reliability In Psychology

In psychology, validity and reliability are fundamental concepts that assess the quality of measurements.

• Validity refers to the degree to which a measure accurately assesses the specific concept, trait, or construct that it claims to be assessing. It refers to the truthfulness of the measure.
• Reliability refers to the overall consistency, stability, and repeatability of a measurement. It is concerned with how much random error might be distorting scores or introducing unwanted “noise” into the data.

A key difference is that validity refers to what’s being measured, while reliability refers to how consistently it’s being measured.

An unreliable measure cannot be truly valid because if a measure gives inconsistent, unpredictable scores, it clearly isn’t measuring the trait or quality it aims to measure in a truthful, systematic manner. Establishing reliability provides the foundation for determining the measure’s validity.

A pivotal understanding is that reliability is a necessary but not sufficient condition for validity.

It means a test can be reliable, consistently producing the same results, without being valid, or accurately measuring the intended attribute.

However, a valid test, one that truly measures what it purports to, must be reliable. In the pursuit of rigorous psychological research, both validity and reliability are indispensable.

Ideally, researchers strive for high scores on both -Validity to make sure you’re measuring the correct construct and reliability to make sure you’re measuring it accurately and precisely. The two qualities are independent but both crucial elements of strong measurement procedures.

Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the beck depression inventory The Psychological Corporation. San Antonio , TX.

Clifton, J. D. W. (2020). Managing validity versus reliability trade-offs in scale-building decisions. Psychological Methods, 25 (3), 259–270. https:// doi.org/10.1037/met0000236

Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10 (4), 255–282. https://doi.org/10.1007/BF02288892

Hathaway, S. R., & McKinley, J. C. (1943). Manual for the Minnesota Multiphasic Personality Inventory . New York: Psychological Corporation.

Jannarone, R. J., Macera, C. A., & Garrison, C. Z. (1987). Evaluating interrater agreement through “case-control” sampling. Biometrics, 43 (2), 433–437. https://doi.org/10.2307/2531825

LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11 (4), 815–852. https://doi.org/10.1177/1094428106296642

Watkins, M. W., & Pacheco, M. (2000). Interobserver agreement in behavioral research: Importance and calculation. Journal of Behavioral Education, 10 , 205–212

Related Articles

Research Methodology

Qualitative Data Coding

What Is a Focus Group?

Cross-Cultural Research Methodology In Psychology

What Is Internal Validity In Research?

Research Methodology , Statistics

What Is Face Validity In Research? Importance & How To Measure

Criterion Validity: Definition & Examples

Reliability and validity: Importance in Medical Research

Affiliations.

• 1 Al-Nafees Medical College,Isra University, Islamabad, Pakistan.
• 2 Fauji Foundation Hospital, Foundation University Medical College, Islamabad, Pakistan.
• PMID: 34974579
• DOI: 10.47391/JPMA.06-861

Reliability and validity are among the most important and fundamental domains in the assessment of any measuring methodology for data-collection in a good research. Validity is about what an instrument measures and how well it does so, whereas reliability concerns the truthfulness in the data obtained and the degree to which any measuring tool controls random error. The current narrative review was planned to discuss the importance of reliability and validity of data-collection or measurement techniques used in research. It describes and explores comprehensively the reliability and validity of research instruments and also discusses different forms of reliability and validity with concise examples. An attempt has been taken to give a brief literature review regarding the significance of reliability and validity in medical sciences.

Keywords: Validity, Reliability, Medical research, Methodology, Assessment, Research tools..

Publication types

• Biomedical Research*
• Reproducibility of Results

Types of Validity: What Every Researcher Should Know

Not sure if your test really measures what it’s supposed to? Let’s explore the different types of validity.

Validity in research ensures that the conclusions drawn from a study accurately represent the phenomena being studied. Without validity, there’s a risk that findings may be inaccurate or misleading, leading to flawed interpretations and potentially misguided actions or decisions. Validity helps researchers confirm that their measurements and observations truly capture what they intend to measure, thus enhancing the reliability and trustworthiness of their research. Also, valid research enables us to have confidence in the knowledge generated and its applicability to real-world situations, making it a cornerstone of scientific inquiry and decision-making processes. This article explores the general concept of validity in research, including some of the different types and significance in various research contexts.

What Is Validity?

Validity is the degree to which a study accurately measures what it purports to measure or the extent to which its findings are dependable and meaningful. It ensures that the conclusions drawn from a study are trustworthy and applicable to the real world. This kind of study is essential because it underpins the credibility and reliability of research outcomes. Without validity, researchers cannot confidently claim that their findings accurately represent the phenomena under investigation or that their conclusions are applicable beyond the specific context of the study. 

To elevate the credibility and relevance of their conclusions, researchers should meticulously align measurements with intended constructs and carefully consider contextual influences on observations. Establishing validity involves careful design, execution, and interpretation of research, incorporating various strategies to ensure that the study accurately captures the intended constructs or relationships. Validity distinguishes between high-quality and poor-quality research reports, underscoring the importance of rigorously testing and enhancing validity and reliability in research endeavors. There are some types of validity to consider when evaluating the quality of a measurement instrument or assessment tool.

Read more: Reliability vs Validity in Research: Measuring What Matters

Face Validity

Face validity is the extent to which a measurement or assessment appears, on the surface, to measure what it is intended to measure. It is a subjective judgment based on the face value of the measurement instrument or assessment tool, rather than on empirical evidence or statistical analysis. It evaluates whether a test or measure “appears to” assess the construct or concept it purports to measure. Although it doesn’t provide definitive evidence of a measure’s validity, it can serve as an initial indicator of its potential usefulness.

Examples Of Face Validity

1. A questionnaire designed to assess job satisfaction includes items such as “I enjoy my work” and “I feel valued by my employer.” These items appear to measure job satisfaction based on their face value.

2. A depression screening tool consists of questions about feelings of sadness, hopelessness, and loss of interest in activities. The content of the tool appears to be relevant to assessing symptoms of depression.

3. A teacher evaluation form includes criteria such as knowledge of the subject matter, clarity of instruction, and availability of student support. These criteria seem to align with the aspects of teaching effectiveness.

Strengths And Weaknesses Of Face Validity

Content validity.

Content validity assesses the extent to which the items or questions included in a measurement instrument adequately represent the entire range of content or construct being measured. It assesses whether the measurement instrument comprehensively covers the domain of interest. Content validity is concerned with the relevance, representativeness, and comprehensiveness of the items or questions included in a measure. Essentially, it evaluates whether the content of the measurement instrument aligns with the construct or concept it intends to measure.

Examples Of Content Validity

1. Academic Test: A teacher develops a test to assess students’ knowledge of a particular subject, such as mathematics. To establish content validity, the teacher ensures that the test questions cover all relevant topics and learning objectives outlined in the curriculum.

2. Employee Performance Evaluation: A company creates an employee performance evaluation form to assess various job-related competencies, such as communication skills and problem-solving abilities. To demonstrate content validity, the company ensures that the evaluation criteria encompass all essential aspects of job performance relevant to the position.

3. Health Assessment Questionnaire: A healthcare provider designs a questionnaire to assess patients’ overall health status, including physical, mental, and emotional well-being. To establish content validity, the healthcare provider ensures that the questionnaire items cover a wide range of health-related domains, such as symptoms, lifestyle factors, and quality of life indicators.

Strengths And Weaknesses Of Content Validity

Construct validity.

Construct validity evaluates how closely a measurement or assessment reflects the underlying theoretical concept or construct it intends to measure. It assesses whether the operationalization of variables effectively captures the underlying theoretical concepts or constructs. In other words, construct validity evaluates whether the measurements or indicators used in a study truly reflect the abstract ideas or constructs being studied. It is essential to ensure that researchers are measuring what they intend to measure and drawing valid conclusions from their research.

Examples Of Construct Validity

1. Intelligence Test: A researcher develops a test to measure intelligence. To establish construct validity, the researcher may administer the test to a diverse group of participants and compare their test scores with other measures of intelligence, such as academic performance or IQ tests.

2. Personality Inventory: A psychologist creates a questionnaire to assess personality traits, such as extraversion and conscientiousness. To demonstrate construct validity, the psychologist may conduct factor analysis to confirm that the questionnaire items load onto the expected personality dimensions.

3. Depression Scale: A clinician designs a scale to measure symptoms of depression. To establish construct validity, the clinician may administer the scale to individuals diagnosed with depression and compare their scores with individuals without depression to ensure discriminant validity.

Strengths And Weaknesses Of Construct Validity

Criterion validity.

Criterion validity refers to the extent to which the scores obtained from a measurement instrument correlate with scores obtained from other established measures or criteria that assess the same construct. It assesses whether the measurement instrument accurately predicts or correlates with an external criterion. Criterion validity is concerned with demonstrating that the scores obtained from a new measurement instrument are consistent with scores obtained from well-established measures or criteria that assess the same construct or concept.

Examples Of Criterion Validity

Predictive Validity of a College Entrance Exam: A university admissions office administers a standardized entrance exam to prospective students. To establish predictive validity, the admissions office tracks the academic performance of admitted students during their first year of college and examines the correlation between their entrance exam scores and their subsequent grades.

Concurrent Validity of a Depression Scale: A researcher develops a new scale to assess symptoms of depression and administers it to a sample of individuals diagnosed with depression. To demonstrate concurrent validity, the researcher compares the scores obtained from the new scale with scores obtained from an established measure of depression, such as the Beck Depression Inventory, administered concurrently to the same participants.

Strengths And Weaknesses Of Criterion Validity

There are more types of validity, including Internal Validity and External Validity, each crucial for robust research outcomes. For further insights, refer to authoritative sources like ScienceDirect: Internal Validity and External Validity . 

Science Figures, Graphical Abstracts, And Infographics For Your Research

Mind the Graph platform helps scientists by providing them with science figures, graphical abstracts, and infographics for their research. These tools make it easier for researchers to illustrate their experiments, analyze data, and present results clearly and engagingly. With graphical abstracts, scientists can summarize their research articles visually, while infographics offer a simple and effective way to share complex scientific information with others. 

Subscribe to our newsletter

Exclusive high quality content about effective visual communication in science.

Unlock Your Creativity

Create infographics, presentations and other scientifically-accurate designs without hassle — absolutely free for 7 days!

Content tags

Developing an Indicator for Age-Friendly Communities: The Japan Gerontological Evaluation Study

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• For correspondence: [email protected]
• Info/History
• Supplementary material
• Preview PDF

Background Age-friendly communities (AFCs) aim to create inclusive societies for older people. Despite the World Health Organization (WHO)’s emphasis on incorporating dementia-friendliness across all phases, including planning, implementation, monitoring, evaluation, and scale-up, there are very few community-level indicators that incorporate dementia-friendly elements.

Objective To develop a community-level AFC indicator based on WHO AFC guidelines incorporating dementia-friendly elements, and examine its validity and reliability.

Design A repeated cross-sectional study using data from the 2016 and 2019 waves of the Japan Gerontological Evaluation Study.

Setting and Subjects Data were collected from 61 school districts in 16 Japanese municipalities, involving 45,162 individuals aged ≥65 years in 2016, and 39,313 in 2019. The 2016 and 2019 datasets were the development and retest samples, respectively.

Methods After identifying 23 candidate items according to the WHO AFC guidelines and expert reviews, data were aggregated by school district. Exploratory factor analysis on the 2016 data helped derive factor structure, confirming reproducibility with the 2019 data. Internal consistency and test-retest reliability were evaluated.

Results The final 17-item indicator comprised three subscales: Social inclusion and dementia-friendliness (7 items, α = 0.86), Social engagement and communication (5 items, α = 0.78), and Age-friendly physical environment (5 items, α = 0.82). The structure showed adequate test-retest reliability (r = 0.71–0.79; ICC = 0.67–0.78).

Conclusions A valid and reliable 17-item community-level indicator was developed, which aligns with the WHO framework and also incorporates dementia-friendly elements. This indicator is useful for monitoring and evaluating to promote the AFC and dementia-friendly communities.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This study used data from JAGES (the Japan Gerontological Evaluation Study). This study was supported by Grant-in-Aid for Scientific Research (20H00557, 20K10540, 21H03153, 21H03196, 21K17302, 21K11108, 21H00792, 22H00934, 22H03299, 22K04450, 22K13558, 22K17409, 23H00449, 23H03117), Health Labour Sciences Research Grants (19FA1012, 19FA2001, 21FA1012, 22FA2001, 22FA1010, 22FG2001), Research Institute of Science and Technology for Society (JPMJOP1831, JPMJRX21K6) from the Japan Science and Technology (JST), a grant from Japan Health Promotion & Fitness Foundation, TMDU priority research areas grant and National Research Institute for Earth Science and Disaster Resilience. The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the respective funding organizations.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Ethical Committee of Chiba University (approval no. M10460)

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Data Availability

The data underlying this study are from the JAGES and contain sensitive information. Data for research purposes is available upon request. Requests for the JAGES data can be made to dataadmin.ml{at}jages.net .

View the discussion thread.

Supplementary Material

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One

Subject Area

• Gastroenterology
• Addiction Medicine (324)
• Allergy and Immunology (632)
• Anesthesia (168)
• Cardiovascular Medicine (2400)
• Dentistry and Oral Medicine (289)
• Dermatology (207)
• Emergency Medicine (381)
• Endocrinology (including Diabetes Mellitus and Metabolic Disease) (850)
• Epidemiology (11797)
• Forensic Medicine (10)
• Gastroenterology (705)
• Genetic and Genomic Medicine (3768)
• Geriatric Medicine (350)
• Health Economics (637)
• Health Informatics (2408)
• Health Policy (940)
• Health Systems and Quality Improvement (905)
• Hematology (342)
• HIV/AIDS (787)
• Infectious Diseases (except HIV/AIDS) (13347)
• Intensive Care and Critical Care Medicine (769)
• Medical Education (369)
• Medical Ethics (105)
• Nephrology (401)
• Neurology (3524)
• Nursing (199)
• Nutrition (529)
• Obstetrics and Gynecology (680)
• Occupational and Environmental Health (667)
• Oncology (1834)
• Ophthalmology (538)
• Orthopedics (221)
• Otolaryngology (287)
• Pain Medicine (234)
• Palliative Medicine (66)
• Pathology (447)
• Pediatrics (1037)
• Pharmacology and Therapeutics (426)
• Primary Care Research (424)
• Psychiatry and Clinical Psychology (3189)
• Public and Global Health (6184)
• Radiology and Imaging (1290)
• Rehabilitation Medicine and Physical Therapy (751)
• Respiratory Medicine (832)
• Rheumatology (380)
• Sexual and Reproductive Health (373)
• Sports Medicine (324)
• Surgery (405)
• Toxicology (50)
• Transplantation (172)
• Urology (147)

Enhancing student engagement through emerging technology integration in STEAM learning environments

• Open access
• Published: 24 May 2024

Cite this article

You have full access to this open access article

• Mirjana Maričić   ORCID: orcid.org/0000-0001-8447-7735 1 &
• Zsolt Lavicza   ORCID: orcid.org/0000-0002-3701-5068 2  

Emerging technologies can potentially transform education through student engagement. The aim of our study is threefold. Firstly, we aspired to examine the validity and reliability of Reeve and Tsengs’ 4-construct (emotional, behavioral, cognitive, and agentic) engagement scale (EBCA scale). Secondly, we aimed to examine whether and to what extent the integration of emerging technology through virtual simulations (VS) in STEAM activities can improve students’ perceived engagement. Thirdly, we strived to examine how the order of integration of VS in STEAM activities affects students’ perceived engagement. A cross-over research design was used. 84 primary school students (9–10 years old) were assigned to one of the following conditions: STA (science + technology + art); SA (science + art); STA + SA; and SA + STA. The results showed that the 4-factor EBCA scale model is aligned and fits the overall sample well. It was also observed that the longer students are involved in STEAM activities, the better their perceived engagement is, and the more they work on VS, the more they develop the values of attentive listening, directing attention, and investing effort in learning. The order of integration of VS affects perceived engagement, and students who learn with them first perceive engagement better. One of the implications of our study is to examine the metric characteristics of the EBCA scale on different samples as well. Other recommendations are stated in the discussion.

Avoid common mistakes on your manuscript.

1 Introduction

Emerging technologies could have the potential to transform the education system and are currently considered one of the most engaging ways for students to learn the content of various scientific disciplines (Anđić et al., 2024 ; Chen & Chu, 2024 ; Leavy et al., 2023 ; Moreno-Guerrero et al., 2021 ). The role of emerging technologies within STEAM education is not fully understood due to the lack of quality theoretical frameworks, practitioner knowledge, and empirical evidence in educational research (Leavy et al., 2023 ). In particular, little is known about the potential of integrating certain emerging technologies, like virtual simulations (VS), into STEAM learning environments (Thisgaard & Makransky, 2017 ). A limited number of studies found on this topic dealt with determining the contribution of VS, mainly to the development of the following variables: student achievement, scientific inquiry, reasoning and scientific process skills, interest, goals toward STEM-related careers, STEM awareness, and students’ perceptions of STEM activities (D’Angelo et al., 2014 ; Sarı et al., 2020 ; Thisgaard & Makransky, 2017 ). These studies, as well as the meta-analysis by the authors Perignat and Katz-Buonincontro ( 2018 ), which analyzed 44 studies on the topic of identifying the intention of STEAM approaches, singled out the promotion of student engagement as a basic feature of STEAM. Student engagement largely determines all other teaching and learning outcomes. However, this variable was not tested in the mentioned studies but was observed as an indirect construct without determining the method of its measurement. Through indirect observation, it was noticed that the development of student engagement is influenced by instructional practice, the structure of lectures, as well as the interactions of participants in the teaching process (Nicol & Macfarlane-Dick, 2006 ; Wang et al., 2015 ). Given the limited amount of research on this topic and certain methodological ambiguities, it remains important that educators communicate their classroom teaching experiences regarding student engagement (Barlow & Brown, 2020 ). It is particularly recommended to consider factors relevant to the development of this variable (such as instructional practice and content delivery methods) as well as instruments that will directly measure student engagement within STEAM learning environments (Barlow & Brown, 2020 ). Identified research gaps (1. a limited number of studies as well as a lack of understanding of the role and potential of certain emerging technologies, like VS within STEAM education; 2. a lack of research on this topic within which student engagement was measured directly with an appropriate instrument) served as the basis for the implementation of our research. For these purposes, we selected primary school students and the emotional, behavioral, cognitive, and agentic EBCA scale for assessing student engagement as a 4-component construct developed by the authors Reeve and Tseng ( 2011 ). Given that this instrument is intended for students of higher education, we had to modify or adapt it to the needs of our study and check its metric characteristics. With this in mind, we have set a threefold aim. Firstly, we aspired to examine the validity and reliability of the EBCA engagement scale. Secondly, we aimed to examine whether and to what extent the integration of VS in STEAM activities can improve students’ perceived engagement. Thirdly, we strived to examine whether and how the order of integration of VS in STEAM activities affects students’ perceived engagement.

2 Emerging technology integration into STEAM environment

According to the latest reports (Leavy et al., 2023 ), emerging technologies are defined as tools or software that could have the potential to radically transform the current state of education and thus enable more creative and engaging ways of learning and teaching (Leavy et al., 2023 ; Sosa et al., 2017 ). According to this, the following technologies are currently considered “emerging” for education: artificial intelligence (AI), big data, learning analytics, immersive technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), virtual labs and simulations (VS), serious games, robotics, the internet of things, hardware with sensors, wearable devices, and drones (Leavy et al., 2023 ). Integrating emerging technologies into STEAM activities is considered one of the most engaging ways for students to learn content from various scientific disciplines (Anđić et al., 2022, 2023 ; Janković et al., 2023 ; Moreno-Guerrero et al., 2021 ). The ‘T’ letter in the STEAM acronym is primarily used for solving various engineering challenges, programming, or designing computer graphics. These activities aim to research and design active, innovative, creative solutions and create artifacts (digital or otherwise) by balancing technical expertise with artistic vision and expressing knowledge and skills in the global world (Glancy, 2014 ; Jones, 2014 ). Its value, as well as the value of using technology, primarily has long-term aspirations. They are reflected in the transition from a consumer to a producer society, i.e., training the young generations to provide socio-technical contributions using simpler examples in the STEAM classroom and, in the future, complex and useful ones for society (Boy, 2013 ). Within the STEAM approach, the shift from knowledge consumption to production via emerging technology-enabled tools in a collaborative environment is a way for students to contribute to the closer community, learn from each other, and acquire skills in these areas that are necessary for the future (Boy, 2013 ).

In a recent meta-analysis, Leavy et al. ( 2023 ) reviewed and analyzed 43 qualitative empirical studies on the topic of identifying emerging technologies that are used to strengthen STEAM education. These papers are classified into the following categories: (1) AR/VR/MR; (2) Programming and Robotics; (3) Maker Movement; and (4) Other Technological Applications. The general importance of the integration of the mentioned emerging technologies into the STEAM approach is reflected in the development of 21st -century skills such as creativity, persistence, and problem solving, attitudes towards computing, creative thinking, and learning in this way through promoting engagement, collaborative problem solving, hands-on learning experiences, and providing strong motivation to promote equity (Leavy et al., 2023 ). However, the authors state that this meta-analysis is limited in providing insight into how emerging technologies can transform and influence learning due to the lack of quality theoretical frameworks, practitioner knowledge, and empirical evidence. Also, bearing in mind that this field is developing, there is a possibility that not all examples of good practice are included due to poor dissemination and recording of the results. Furthermore, we will list a few examples of papers that were not included in this meta-analysis.

In a study by Laut et al. ( 2015 ), STEM activities are empowered with robotics to develop and understand the connection between biology and engineering. The students were given the task of completing their biomimetic robotic fish through STEM and letting the finished product be verified at the New York Aquarium to observe the fish’s response in biology. The results of this study showed that robotics can strengthen STEM learning and contribute to students’ understanding of the connection between these two disciplines. In the study by Techakosit and Nilsook ( 2018 ), the contribution of the integration of AR within STEM activities to the development of STEM literacy was examined. The results showed that, for learning the STEM contents, imagination, design ability, finding information, and using STEM’s basic abilities to solve problems were very important skills that students develop while learning in this way. Further, Chen and Huang ( 2020 ) investigated the contribution of serious game-based learning to the strengthening of STEAM activities from the aspect of improving achievement and reducing the cognitive load when working with primary school students (13–14 years old). The results of this study showed that game-based learning can strengthen STEAM activities and contribute to the development of student achievements and the reduction of cognitive load. Emerging technologies have the potential to initiate inevitable and necessary changes in the educational system through redefining and reshaping teaching that is consistent with STEAM principles (Leavy et al., 2023 ). What is needed to utilize and maximize the opportunities of these technologies within the STEAM approach is a reconceptualization of school programs, teaching, learning, and assessment methods (Meletiou-Mavrotheris, 2019 ). We will further look at the potential of integrating certain emerging technologies into the STEAM approach, i.e., we will focus on the integration of VS in the STEAM learning environment.

3 Integration of virtual simulations into STEAM environment

Virtual simulations (VS) imply computer modeling of reality, i.e., computer-based representations of real or hypothesized scientific phenomena and processes, with which students, in an interactive way in a virtual environment, become familiar with the mental models of scientists and construct their own to understand and explain certain scientific phenomena (Falloon, 2019 ; Sanina et al., 2020 ; Zhang, 2014 ). They offer the possibility of observing scientific processes visible and invisible to the naked eye, as well as the possibility of visualizing abstract, less abstract, and non-abstract concepts (e.g., electrons, molecules, light rays) (Maričić et al., 2023 ; Olympiou et al., 2013 ). The basic intention of this visualization is reflected in the transformation of abstract phenomena, i.e., theoretical-conceptual constructions into perceptual representations, to build a bridge between the students’ understanding of those concepts in the natural environment and the mechanism of their actual functioning (Sanina et al., 2020 ). In addition, VS offers the possibility of simplifying the investigated phenomenon or process by highlighting the target elements being observed and removing complexity, or it can be modified to a simpler or shorter time frame to more easily interpret certain natural phenomena (de Jong et al., 2013 ; Maričić et al., 2023 ). VS appear in the form of computer-based animations such as models, simulations, and experiments (Falloon, 2019 ). All these forms offer students the opportunity to enter a micro-virtual world where they can manipulate virtual equipment, materials, and variables of interest and immediately access the obtained results (Scalise et al., 2011 ; Wen et al., 2020 ). Through virtual models, simulations, and experiments, students can observe and investigate those natural phenomena and processes that are not easy to observe and investigate in real-life circumstances (Zhang, 2014 ). In addition to the above, they can be more manageable, more flexible, safer, more profitable, and faster to implement than real hands-on activities (Wen et al., 2020 ).

When working on VS, students encounter two processes: transformation and regulation. In the process of transformation, students produce direct information by forming hypotheses, designing experiments, and concluding. Through the process of regulation, students connect the variables, conditions, and events presented in the problem, identify key variables, and visualize the conditions of the simulation (Lim, 2004 ; Sarı et al., 2020 ). As a result of these processes, we can notice that VS can play an active role in the STEAM learning environment in terms of supporting the research process and providing modeling opportunities (Sarı et al., 2020 ). These processes can then be carried out through real hands-on activities, which include the integration of other disciplines such as engineering, art, and mathematics. Within them, research support is strengthened through previous manipulation of the phenomenon or process in virtual conditions, visualization of the invisible, simplification of reality, regulation of the time frame, and manipulation of the variable of interest, while the modeling process can be performed more faithfully and creatively through the design of active, innovative, creative solutions using knowledge of engineering and mathematics and the creation of artifacts through balancing real material with artistic vision. Thus, VS can strengthen and support STEAM learning, and students can express their skills and knowledge through different disciplines.

Although VS are considered promising emerging technologies that can support STEAM learning, very little is known about their potential in research practice (Thisgaard & Makransky, 2017 ). In a meta-analysis by D’Angelo et al. ( 2014 ), which dealt with determining the contribution of VS within the STEM approach, 59 studies were reviewed. The results of this meta-analysis showed that VS can strengthen STEM activities in terms of student achievement, scientific inquiry, reasoning skills, and non-cognitive outcomes. Although this meta-analysis showed that VS can strengthen STEM learning, the authors state that it is necessary to carry out more research to gain insight into the benefits of VS within the STEM domain. The research by Thisgaard and Makransky ( 2017 ) examined the contribution of VS to students’ knowledge of evolution, interest, and whether simulations could catalyze STEM academic and career development. High school students (18 years old) were supposed to identify an unknown animal found on the beach through VS while investigating various aspects of natural selection and genetics through video displays of genetic links and 3D visualization of a population of a species on an island. The results of this study showed that VS can strengthen STEM learning in terms of developing student interest and goals toward STEM-related careers. Sarı et al. ( 2020 ) analyzed the contribution of VS within STEM activities to the development of students’ scientific process skills, STEM awareness, and views on activities. Second-year undergraduate students participated in the research. The results showed that VS can strengthen STEM learning from the perspective of these variables and that students believe that STEM activities provide numerous advantages, such as designing and developing engineering products, conducting experiments, and reducing errors.

The contribution of VS was examined within STEM learning, focusing mainly on the following variables: student achievement, scientific inquiry, reasoning and scientific process skills, interest, goals toward STEM-related careers, STEM awareness, and students’ perceptions of STEM activities (D’Angelo et al., 2014 ; Sarı et al., 2020 ; Thisgaard & Makransky, 2017 ). These studies, as well as the meta-analysis by Perignat and Katz-Buonincontro ( 2018 ), which reviewed 44 studies on the topic of STEAM approaches (i.e., identifying the purpose of STEAM education, definitions of STEAM acronyms, and definitions of ‘A’ in STEAM), single out the engagement of students as the basic feature of STEAM education within these disciplines. However, student engagement was not tested in these studies but was observed as an indirect construct without determining the method of its measurement. In the next section, we will focus on this variable.

4 Involvment/engagement theory

The understanding of the concept of student engagement was contributed by Astin ( 1984 ), who studied student development for more than 20 years. Instead of the term engagement, Atkin uses the term involvement and focuses on college students. According to him, student involvement refers to the amount of physical and psychological energy that students devote to the academic experience (Astin, 1984 ). This determination is based on the following five postulates, shown in Fig.  1 .

Postulates of student involvment

Atkin’s definition was later expanded by the director of the National Survey of Student Engagement, George Kuh, who states that engagement, in addition to the investment of physical and mental energy of the participants in the educational process, also represents the effort of the institution that it invests in using an effective educational performance (Axelson & Flick, 2010 ). Later, the determinations of student engagement became more and more complex, taking into account different aspects of education, but what they all have in common is that an educational institution with an educational system is not only a place where acquired knowledge is transferred from individual to individual but also a place where different types of relationships develop. These relationships exist between the participants in the educational process (the social component) as well as between the participants and the learning object (the intellectual component), and they are characterized by a certain emotional flow. Bearing that in mind, according to modern understandings, engagement is defined as a state of emotional, social, and intellectual readiness for learning, which is characterized by curiosity, participation, and the drive to learn more (Abla & Fraumeni, 2019 ). These connections can be observable, like visible behavior, but also unobservable, like internal attitudes. With that in mind, the authors Fredricks et al. ( 2004 ) identified three different types of engagement: emotional, behavioral, and cognitive, while Reeve and Tseng ( 2011 ) described a fourth type: agentic engagement (see Fig.  2 ).

Engagement as a 4-component construct

Therefore, engagement can be defined as a multi-dimensional construct. Within STEAM education, it has been observed that this variable largely determines all other teaching and learning outcomes for students (Barlow & Brown, 2020 ; Hong et al., 2020 ; Khamhaengpol et al., 2021 ). As previously stated, student engagement in these studies was not directly measured but was observed as an indirect construct. Through indirect observation, it was noticed that the development of student engagement is primarily influenced by instructional practice, the structure of lectures (and exams), as well as the interactions of participants in the teaching process (Nicol & Macfarlane-Dick, 2006 ; Wang et al., 2015 ). However, given the limited amount of research on this topic and certain methodological ambiguities within it, it remains important that instructors and educators consider and communicate their practical classroom teaching experiences regarding student engagement (Barlow & Brown, 2020 ). In doing so, it is particularly recommended to take into account factors important for the development of this variable, including the structure of lectures, content delivery methods, and student interactions, as well as instruments that will measure this variable in a direct way within the STEAM learning environments (Barlow & Brown, 2020 ).

5 Purpose of the study

Based on a detailed review of the literature, the following research gaps were identified: (1) a limited number of studies as well as a poor understanding of the role and potential of certain emerging technologies like VS within STEAM education; (2) a lack of research on this topic in which student engagement was directly measured with an appropriate instrument. To fulfill the mentioned research gaps, we decided to conduct this study. For these purposes, we selected primary school students and the scale for measuring emotional, cognitive, behavioral, and agentic engagement—the EBCA scale by Reeve and Tseng ( 2011 ). As the EBCA scale is intended for high school students, we had to modify it, adapt it to the needs of our research, and check its metric characteristics. With this in mind, we have set a threefold aim. Firstly, we aspired to examine the validity and reliability of the EBCA engagement scale. Secondly, we aimed to examine whether and to what extent the integration of VS in STEAM activities can improve students’ perceived engagement. Thirdly, we strived to examine whether and how the order of integration of VS in STEAM activities affects students’ perceived engagement. The following research questions arise from the stated three-fold aim:

Can the EBCA engagement scale be used validly and reliably in the primary school context?

Whether and to what extent the integration of VS in STEAM activities can improve students’ perceived engagement?

Whether and how the order of integration of VS in STEAM activities affects students’ perceived engagement?

6 Methodology

6.1 research design.

The research was carried out according to the cross-over research design (Crowder & Hand, 2017 ; Hughes et al., 2022 ), in which the students of the experimental groups undergo all STEAM (STA and SA) learning conditions but only in a different order. The research design is shown in Fig.  3 .

Research design

This Pre—Post—Post-Delayed engagement assessment design was used to collect measurement outcomes before, during and after the intervention (Craig et al., 2012 ). Such a design allowed us to gain insight into whether and to what extent the integration of VS into STEAM activities can improve students’ perceived engagement, as well as how the order of VS integration in STEAM activities affects students’ perceived engagement.

For this research, schools from the district were recruited, and classes of 3rd -grade students that were available to the researcher were selected. A convenience sampling method was applied. The students in the selected classes were given a pre-engagement scale (PES1) to determine the level of their previous perceived engagement in science classes. Those classes of students who showed an approximate perception of previous engagement in the classes were retained in the research. PES1 was used as one of the criteria for equalizing the groups (and as a covariate in the analysis of the results). Selected classes of students were then randomly assigned to one of the STEAM conditions: STA (science, technology, and art) or SA (science and art). Through the combination of these STEAM conditions and the usage of cross-over design, four groups were formed: two control (C1 and C2) and two experimental (E1 and E2): C1 - STA + STA, C2 - SA + SA, E1 - STA + SA, and E2 - SA + STA. After the formation of the groups, the first lesson was held in C1 and E1 (STA lesson), and C2 and E2 (SA lesson). Then the students were given a post-engagement scale (PES2) to determine the level of their perceived engagement after participating in the first part of the intervention. Next week, the second lesson was held in C1 and E2 (STA lesson) and C2 and E1 (SA lesson). After the end of the second lesson, the students were given a delayed post-engagement scale (PES3) to establish their level of perceived engagement after participating in the second part of the intervention.

6.2 Intervention

For the implementation of STEAM activities, the science content Magnetism was selected. The first lesson included the following concepts: what is a magnet, the shapes of a magnet, the poles of a magnet, the lines of force of a magnetic field, attraction and repulsion, and action through different environments. The second lesson included the following concepts: magnetization, magnetic field strength, natural and artificial magnets (make an artificial magnet), and the effect of magnets in different environments (make a boat). These scientific contents are strengthened and integrated with the contents of art: landscape and abstract art (in the first lesson, abstract art , and the second lesson, landscape ). In addition to the concepts of abstract art and landscape, elements of visual art are also integrated into the lessons to introduce a science concept. These elements included the following: observing works of art, painting examples of abstract art and landscapes, and creating original works of art that also present scientific concepts about magnetism. Through the integration of the content of the sciences and arts with technology, the STA condition was formed. Technology integration referred to the introduction of VS (from the JavaLab series) on magnetism to strengthen the understanding of the scientific concepts of these contents. VS offers the possibility of visualizing those abstract concepts that students cannot see with the naked eye, such as the lines of force of the magnetic field and their behavior during the approaching of the same and different poles of the magnet, the concept of magnetization, the formation of domains within metals, and their orientation. By integrating the content of the sciences and arts (without technology), the SA condition was formed. Basic STEAM conditions are shown in Fig.  3 .

By combining the STA and SA conditions through a cross-over design, two more conditions were formed - STA + SA and SA + STA. STEAM activities will be briefly described below.

6.2.1 STA and SA conditions

All students were introduced to the intervention in the same way. They were told the story of the shepherd Magnus - how the ore magnetite was discovered and how the term magnetism came about. During this conversation, students were shown an example of this ore.

STA condition : Lesson 1 – The students were then shown paintings from the series Magnetic North: Imagining Canada in paintings by seven famous Canadian painters (abstract). Through a conversation with the researcher about the paintings, the displayed techniques, and the fascinating name of the entire collection of these works, they came up with the term magnetism. This term is then connected to the term from the story of the shepherd Magnus. Then, through hands-on activities, the students went over the following concepts: what is a magnet, the shapes of a magnet, the poles of a magnet, the lines of force of a magnetic field, attraction and repulsion, and the action of a magnet in different environments. Through VS from the JavaLab series about magnets, students strengthened their knowledge about magnetism, magnetic fields, magnet poles, and magnetic field lines of force. After this part, students were introduced to the concept abstract art. They were shown paintings by famous abstract artists, such as Clyfford Still. The students discussed the paintings and communicated what impressed them, i.e., what was magnetic about them. After that, with the usage of different art materials and media, the students were placed in a position to create their magnetic abstract work. Then, through the main activity, students had to create an original 2D artwork that integrates elements of science and art. The students painted their abstract work of art with magnets through the property of magnets acting through different environments .

Lesson 2 - The students were shown paintings from the series Magnetic North: Imagining Canada in paintings by Canadian artists, but this time the landscape ones. The researcher introduced the students to the concept of magnetism through a conversation about the paintings, the techniques shown, and the fascinating name of the collection of these works. This concept is connected with the concept from the story of the shepherd Magnus. The students then went through the following concepts through hands-on activities: magnetization, magnetic field strength, natural and artificial magnets (make an artificial magnet), and the effect of magnets in different environments (make a boat). Through VS, students strengthened their knowledge of magnetism and magnetization, magnetic fields, and natural and artificial magnets. After this, students were introduced to the concept of landscape. They were shown paintings by famous landscape painters from the Barbizon School. The students discussed the paintings and communicated what impressed them, i.e., what was magnetic about them. After that, with the usage of different art materials and media, the students were placed in a position to create their own magnetic landscape. Then the main activity was introduced, in which the students had to create an original 3D artwork that integrates elements of science and art. The idea was to create an original 3D interactive landscape—an image of a landscape in which a part of the artwork is integrated, which can be moved by a magnet and make it interactive.

SA condition : This condition included the integration of science and art in both lessons, but without technology, i.e. all those elements (in the same order) from the STA condition were represented here, only without the usage of VS.

STA + SA condition : Within this condition, the first lesson was performed under the STA condition, while the second was carried out under the SA condition (without technology).

SA + STA condition : This condition implied that the first lesson was performed according to the SA condition, while the second was carried out according to the STA condition.

6.3 Sampling

84 3rd -grade students (9–10 years old, M  = 9.643, SD  = 0.482) from two primary schools in Eastern Europe participated in the research. The classes were recruited from schools attended by students with a diverse body: students from national minorities and different ethnic backgrounds, as well as students who learn according to the IEP. In the research, those classes of students that showed an approximate perception of previous engagement on PES1 and those students within those classes who filled out all three PESs were retained. For this research, four classes of 3rd -grade students were recruited and randomly assigned to one of the STEAM conditions. The random distribution in our research was performed so that already-formed classes were randomly assigned to one of the four STEAM conditions (in each of 21 students). Teacher bias was excluded by introducing a trained researcher into the intervention. Including all students in both STA and SA conditions allowed us to monitor the impact of the order of VS integration on students’ perceived engagement.

6.4 Data collection

Data in this research were collected using a previously created instrument, the EBCA scale, for assessing students’ perceived emotional, behavioral, cognitive, and agentic engagement by Reeve and Tseng ( 2011 ). Given that this scale is intended to measure the perceived engagement of high school students, we had to adapt it to the needs of our study to successfully assess the perceived engagement of primary school students. These adaptations were also reflected in the slight modification of the items, which resulted in the creation of three scales: PES1, PES2, and PES3 (for example, on PES1, the items are directed to the state before the implementation of the intervention, on PES2, the items are directed to the state immediately after the first part of the intervention, and on PES3, the items are directed to the state after the implementation of the intervention). Before conducting the research, permission was requested from the author to adapt the scale. Adaptation resulted in several rounds of revision in which some items were excluded. During this process, experts in the field of methodology were consulted, as well as teachers with work experience spanning over 10 years, as the first assessors of the validity of the scale. The revised scale was adapted for 84 primary school students, and for the second round of checking construct validity and reliability, confirmatory factor analysis (CFA) was performed. The scale consists of four blocks, of which the emotional block has four items, the behavioral block has five items, the cognitive block has five items (the original has eight, i.e., three items from this block were excluded), and the agentic block has five items. These items are intended to assess four different types of students’ perceived engagement. Within emotional engagement, the following values were monitored: enjoyment, fun, interest, and curiosity. As part of the behavioral engagement, the following values are followed: careful listening, paying attention, trying hard, careful listening about new topics, and trying hard when starting something new. Within cognitive engagement, the following values were monitored: relating to prior knowledge, relating to personal experience, connecting different ideas into a meaningful whole, creating own examples to understand the concepts, and reviewing what was done. Within agentic engagement, these values are followed: asking questions to make the class active and lively; informing the teacher about personal interests; informing the teacher about the need to improve achievement; informing the teacher about preferences; and suggesting ideas for class improvement. The obtained results for each type of engagement, as well as the discussion about them, will be shown in the next two sections, but in such a way that these data follow each part of our threefold aim.

7.1 First part: Construct validity and reliability of the EBCA scale

The skewness and kurtosis values for PES1, PES2 and PES3 are between -2 and + 2 which shows that the data is normally distributed (Byrne, 2010 ; Hair et al., 2010 ). Kaiser-Meyer-Olkin (KMO) and Bartlett’s Test of Sphericity tests were used to determine the suitability of the data for confirmatory factor analysis (CFA). The KMO and Bartlett’s Test of Sphericity values for PES1, PES2 and PES3 were found to be statistically significant ( p  < .000). It was ensured that the sample size was sufficient for data analysis (Tabachnick & Fidell, 2007 ) (see Table  1 ).

The obtained values were accepted as an indication that CFA could be performed. IBM SPSS AMOS program was used for CFA. In the upcoming paragraphs, we will present the CFA results for each scale.

Within CFA results, we monitored the values of various fit indices, which are primarily used to assess the fit of the model to the data. As a result of the analysis conducted on 19 items, the RMSEA values for PES1, PES2 and PES3 were found to be within acceptable range. Fit indices for PES1 show that this scale fits the overall sample well (χ 2 (140, N  = 84) = 183.437, p  = .008; CFI = 0.977, TLI = 0.972, RMSEA = 0.061, SRMR = 0.076). Covariance of error terms based on modification indices (MI > 20) was created for six pairs which improved the model. The final model is shown in Fig.  4 .

STA and SA conditions

Convergent validity and composite reliability (CR) of PES1 are also good. All factor loadings have a value above 0.60 (Fig.  4 ). Average variance extracted (AVE) values are above 0.05, and CR values are above 0.70 for all constructs (Hair et al., 2017 ). Cronbach alpha (CA) values are also above 0.70. Discriminative validity of the scale is good - the square root of AVE values (bold values) are higher than inter-variable values (below bold values) (Fornell & Larcker, 1981 ) (Table  2 ).

Fit indices for PES2 show that this scale fits the overall sample also well (χ 2 (146, N  = 84) = 197.611, p  = .003, CFI = 0.939, TLI = 0.929, RMSEA = 0.065, SRMR = 0.065) (Fig.  5 ).

Measurement model of PES1

Convergent validity and composite reliability (CR) of PES2 are also good. Factor loadings are above 0.60 (Fig.  5 ), AVE values are above 0.05, CR and CA values are above 0.70 for all constructs. The discriminative validity of the scale is good - the square root of AVE values is higher than inter-variable values (Table  3 ).

Fit indices for PES3 show that this scale fits the overall sample well (χ 2 (145, N  = 84) = 189.682, p  = .007, CFI = 0.948, TLI = 0.939, RMSEA = 0.061, SRMR = 0.064). Covariance of error terms based on modification indices (for one pair - MI > 20) was created for one pair which improved the model. The final model is shown in Fig.  6 .

Measurement model of PES2

Convergent validity and composite reliability (CR) of PES3 are also good. Factor loadings are above 0.60 (Fig.  6 ), AVE values are above 0.05, CR and CA values are above 0.70 for all constructs. The discriminative validity of the scale is good - the square root of AVE values is higher than inter-variable values (Table  4 ).

Since the data showed a normal distribution, parametric tests were used for further analysis. A repeated measures ANOVA was used to determine the difference in student-perceived engagement between the three different time points. The ANOVA and ANCOVA analysis where used to determine whether there was a difference in the students’ perceived engagement between different STEAM conditions at PES1, PES2 and PES3. An independent t-test was used to determine whether there was a difference in the order of VS integration. These analyzes cover the second and third parts of the aim Fig.  7 .

Measurement model of PES3

7.2 Second part - contribution of the VS in STEAM activities

One-factor ANOVA analysis of repeated measures compared the difference in students’ perceived engagement between three different time points - PES1, PES2 and PES3. The results of this analysis for all groups indicate a significant influence of time for all types of engagement, i.e. that the level of perceived emotional, behavioral, cognitive and agentic engagement changed significantly during these three-time points (Table  5 ).

These differences were further processed, to establish between which time points within each type of engagement there was a significant difference. These results are shown in (Table  6 ).

Based on these results, it can be observed that there are significant differences between PES1 and PES2, as well as PES1 and PES3 within each type of engagement, while significant differences between PES2 and PES3 exist within behavioral, cognitive and agentic engagement.

Further analyses considered the differences between the groups. ANOVA analysis found that there was no significant difference in PES1 in terms of perceived engagement ( F (3, 80) = 0.484, p  = .695, C1 M  = 3.386, SD  = 0.299; C2 M  = 3.429, SD  = 0.375; E1 M  = 3.470, SD  = 0.405; E2 M  = 3.512, SD  = 0.320). PES1 scores served as a covariate for ANCOVA analysis.

ANCOVA analysis found that there was a significant difference on PES2 regarding perceived engagement ( F (3, 79) = 6.980, p  = .000, ηp 2  = 0.210, covariate under control F (3, 79) = 27.407, p  = .000, ηp 2  = 0.258). Through further analysis, we tried to determine within which type of engagement and between which groups this difference exists. The results showed that there was a difference in behavioral engagement ( F (3, 79) = 3.835, p  = .013, ηp 2  = 0.127, covariate controlled F (3, 79) = 94.359, p  = .000, ηp 2 =. 544) between the STA and SA condition ( p  = .024) and the STA and SA + STA condition ( p  = .034), where the students of the C1 group ( M  = 4.324, SD  = 0.171) showed significantly better results compared to the students of C2 ( M  = 4.076, SD  = 0.462) and E2 groups ( M  = 4.124, SD  = 0.449).

ANCOVA analysis found that there was a significant difference in PES3 regarding perceived engagement ( F (3, 79) = 7.977, p  = .000, ηp 2  = 0.233, controlled covariate F (3, 79) = 19.732, p  = .000, ηp 2  = 0.200). Through further analysis, we tried to determine within which type of engagement and between which groups this difference exists. The results showed that there was a difference in terms of behavioral engagement ( F (3, 79) = 5.031, p  = .003, ηp 2  = 0.160, covariate under control F (3, 79) = 82.300, p  = .000, ηp 2 =. 510) between the STA and SA conditions ( p  = .003), where the students of the C1 group ( M  = 4.419, SD  = 0.374) showed significantly better results compared to the C2 students ( M  = 4.124, SD  = 0.403).

7.3 Third part - contribution of the VS integration order

The results of independent t-test showed that the p -value is close to the significance threshold t (40) = 1.753, p  = .087, but does not exceed it. The students of the E1 group ( M  = 4.279, SD  = 0.170) showed better results compared to the students of the E2 group ( M  = 4.183, SD  = 0.184), which indicates that the VS integration in the first part of STEAM intervention contributes to a greater extent to the development of student perceived engagement compared to integration of VS in the second part of STEAM intervention.

8 Discussion

8.1 first part: construct validity and reliability of the ebca scale.

The results of our research show that the 4-factor engagement scale model is aligned, i.e., it fits the overall sample well. It should be noted that for the PES1 and PES3 scales, the covariance of the error term was created based on the modification indices for some pairs, which further improved their model fit. In view of this, it is suggested to check the fitness of the model on another sample and, if necessary, make modifications or remove certain items from the scale. Similar results were found in the research of Ritoša et al. ( 2020 ), in which the model fit of three constructs of engagement was checked: emotional, behavioral, and cognitive (the construct of emotional disaffection was added to the scale) on a sample of students from preschool (ages 6–7). After the modifications based on the modification indices, this scale showed a good fit. The engagement scale with all four constructs was tested in the research of Maričić et al. ( 2023 ) on a sample of primary school students (10–11 years old) and in the research by Zainuddin et al. ( 2020 ) on a sample of students from secondary school (16 years old), but regardless of the modifications that were made by the needs of the research, the model fit was not checked. The model fit of the original scale was checked by authors Reeve and Tseng ( 2011 ) on a sample of students from high school (over 16 years old). The 4-factor model proved to be adequate. Regarding the convergent validity, reliability, and discriminative validity of the engagement scales (PES1, PES2, and PES3), good results were obtained in our study, and this shows that the engagement scale in this form can be validly and reliably used in an educational context when working with primary school students (ages 9–10). Similar results were observed in the research of Ritoša et al. ( 2020 ). It is important to indicate that our results are limited in terms of generalization because the model fit was checked on a smaller sample of students aged 9–10 from Eastern Europe. The modified EBCA scale should be tested in work with students of different grade levels and from different ethnic and cultural backgrounds, which will improve the generalization of the results and affect their applicability on a more global level.

8.2 Second part - contribution of the VS in STEAM activities

The results further show that the level of perceived emotional, behavioral, cognitive, and agentic engagement changes significantly over time, i.e., the longer students are involved in STEAM activities, the better their perceived engagement is. As noticed in previous studies through indirect observation, the STEAM approach can enhance student engagement (Hong et al., 2020 ; Khamhaengpol et al., 2021 ). Our study deepens these observations as it provides results generated as a product of direct measurement of this variable. Observed differences are greatest within agentic, then emotional, behavioral, and finally cognitive engagement between all three time points. These observations are consistent with observations from previous studies indicating that agentic engagement offers great potential in terms of enhancing learning (Reeve & Tseng, 2011 ). Students of all groups perceived this type of engagement the best over time because, during the intervention, an atmosphere was created in which they were free to ask questions, express their opinions, follow their interests, and make suggestions. Agentic engagement is proactive, intentional, and purposeful; it offers opportunities to enrich the learning process by making it more personal, interesting, challenging, and valuable for students; and it develops a constructive contribution to the planning and flow of teaching activities in which students have a say. In order to develop this type of engagement, teachers should provide students with autonomy support, i.e., they need to create classroom conditions in which students feel free to express opinions, pursue interests, and ask questions (Maričić et al., 2023 ; Reeve & Tseng, 2011 ). STEAM activities offer that possibility and leave enough space for an optimal level of personalization of the learning process by students, which is very important for improving their perceptions of learning. Our results indicate that the longer the students were engaged in STEAM activities, the more they developed the values of actively asking questions, communicating their interests, the need to improve achievement as well as suggestions for improving learning, the feeling of enjoyment, fun, interest, curiosity, and finally the values of careful listening, focus, and investing effort. In previous research, it was shown that teachers who work with students from higher schools to a significantly greater extent (disproportionately) activate the components of cognitive engagement, while teachers who work with students of lower school age to a significantly greater extent (disproportionately) activate the components of behavioral engagement (Greene et al., 2004 ; Reeve & Tseng, 2011 ). The results of our study are not in line with the aforementioned because it was shown that our STEAM activities in students over time activate the components of all four types of engagement so that none of them is disproportionate to the others. Also, when we compare them, we notice that the agentic, behavioral, and emotional components are only slightly more activated over time than the cognitive ones. A similar pattern was observed in the research of Ritoša et al. ( 2020 ), where preschool children showed a higher level of emotional, behavioral, and cognitive engagement, but in approximate proportions. This is most likely related to the nature of the STEAM activities and the first student participation in them, where the other three types of engagement slightly prevailed. This problem should be further and more deeply examined in future studies.

In addition to the above, the results of our research show that the integration of VS into STEAM activities over time significantly contributes to the development of students’ perceived engagement compared to STEAM activities without technology (SA condition). Similar results were observed in the meta-analysis by Leavy et al. ( 2023 ), in which it was stated that emerging technologies have the potential to increase student engagement, as well as in the study by Katyara et al. ( 2023 ), in which it was shown that the integration of different technologies into learning activities can enrich this process and significantly increase different types of student engagement. Over time, in our study, emerging technology primarily encouraged the development of agentic, behavioral, emotional, and finally cognitive engagement. This shows us that the implementation of VS develops the values of personalization, enrichment of content, and learning conditions, then the values of participation in activities, attention to tasks, investment of effort, perseverance, and absence of behavioral problems, and finally the feeling of joy, fun, interest, and curiosity in the students. Kahu et al. ( 2015 ) found that positive emotions associated with the topic, such as interest, fun, and enthusiasm, come from learning that is integrated with life experience, as well as the intersection between learning materials and students’ work and experience. STEAM’s technology-enhanced approach offers it all. Considering that the research on this topic is limited, it is recommended to investigate this issue more deeply and further through a longitudinal study, which can provide significant insights into the contribution of emerging technologies to the development of different types of student engagement over a longer period of time. These data would indicate the potential of emerging technologies in maintaining student engagement as well as in the development of different types of student engagement, considering the time frame and acquired experience in STEAM activities.

If we consider the results obtained by comparing all four different conditions and groups (while eliminating the time factor), we can also see that STEAM activities enhanced by VS contribute to the development of student-perceived engagement to a greater extent. These differences are significant in terms of behavioral engagement, where it was shown that the constant integration of VS (through both lessons, STA) within STEAM activities significantly contributes to the development of this type of engagement compared to the STEAM condition without VS integration (SA) and the STEAM condition with partial VS integration (only within the second lesson, SA + STA). A similar observation was made in the research by Garcia-Martinez et al. ( 2021 ), in which it was shown that the integration of technology into teaching not only changes the way students learn but also changes their learning behaviors and performance in the long run. Similar results were also observed in the research by Katyara et al. ( 2023 ), where it was noticed that the integration of technology in learning activities contributes to the greatest extent to the development of behavioral engagement. These facts are explained from the perspective of various opportunities and benefits that technologies provide to the development of this type of engagement, such as the following: they make the learners more actively involved in the learning process and encourage them to invest more efforts; reduce the dominance of the teacher; enable students to independently participate in more self-regulating learning activities; therefore, help them to develop self-reliance, persistence, and attention (Katyara et al., 2023 ; Maričić et al., 2023 ; Zinan & Sai, 2017 ). This indicates that students who learn content with STEAM-embedded technology tools develop the values of active involvement, attentive listening, persistence, focus, and investing effort to a much greater extent. These facts can be justified by the benefits that VS offer in terms of learning. While the students were learning through them, they were able to visualize abstract concepts - those that they failed to see through real hands-on experiments such as the lines of force of a magnetic field, their behavior during the approach of the same and different poles of a magnet, the concept of magnetization, the formation of domains within metals, and their orientation, which encouraged them to listen carefully, direct their attention, and put in extra effort when working on VS. Thus, students were significantly more actively involved in STEAM learning activities, which had the greatest impact on the development of behavioral engagement. Such results should be discussed in future research from the Technology Acceptance Model (TAM) theory perspective, which would indicate the extent to which students (and teachers) accept this type of technology as well as their future intentions regarding the usage of VS in teaching. In addition to the above, it is suggested that different types of engagement should be correlated with other variables, such as student achievement and motivation, to see their connection and consider other important components of the teaching process.

8.3 Third part - contribution of the VS integration order

Our results also reveal that the integration of VS at the beginning (in the first STEAM lesson) contributes to a greater extent to the development of students’ perceived engagement compared to the integration of VS at the end (within the second STEAM lesson). Similar results were observed in the research of Hughes et al. ( 2022 ), which examined the order of arts integration within STEAM activities. The results showed that students who studied life and physical science contents first with the integration of art in STEAM activities showed better results compared to those students who studied those contents in a different order. The order of technology integration can be seen as a significant predictor of student engagement in STEAM activities. Students who first learned with STEAM activities in which VS was integrated showed better results in agentic, behavioral, emotional, and cognitive engagement after the first lesson. These data show us that after the first lesson, the students were significantly more enterprising, behaviorally and effectively active, and invested more mental effort in the learning process, which prepared and encouraged them to continue learning about these contents. Also, the integration of another discipline within STEAM activities at the very beginning of the intervention significantly expands the students’ horizons, which leads to multimodal representation of contents, the generation of new ideas, and a more creative approach to learning (Hughes et al., 2022 ). Students learned scientific concepts about magnetism through demonstration, performing real hands-on experiments, and creating original works of art (that present scientific concepts), but also through VS, i.e., through different modalities. This leads us to the potential conclusion that the integration of VS within the first STEAM lesson prepared the students for the initial conceptualization and visualization of abstract concepts, which gave them a valid basis and later facilitated the continuation of learning the same content. These activities particularly influenced the development of agentic and behavioral engagement, i.e., they strengthened the student’s optimal personalization and enrichment of the learning process through participation, attention, effort, and persistence. Given that within the groups, approximate mean values were observed in terms of all types of student engagement, we can note that multimodal representation of contents greatly influenced the development of emotional and cognitive engagement as well, i.e., it stimulated the development of a positive emotional state and cognitive functions in students. This has been demonstrated by several STEAM studies, which confirmed that this approach prepares students for learning and reduces cognitive load (expands the working memory space) because abstract concepts become much more accessible through multiple modalities of representation, which also affects the regulation of conceptual inconsistency (Campbell et al., 2018 ; Maričić et al., 2022a , b ; Wahyuningsih et al., 2020 ). VS offer exactly that possibility - through visualization. Such results should be discussed in future research from the perspective of cognitive load theory, which can shed more light on the contribution of VS to students’ cognitive potentials and their connection with different types of engagement.

9 Conculsion, contribution, implications and limitations

9.1 conclusions.

Based on the analysis of our results, we can conclude that the 4-factor EBCA scale model is aligned and fits the overall sample well, i.e., the engagement scale in this form can be validly and reliably used in an educational context when working with primary school students. STEAM activities can support student-perceived engagement, and the longer students are involved in STEAM activities, the better their perceived engagement is. Over time, this type of learning has the greatest impact on the development of agentic engagement (but not disproportionately compared to other types of engagement). VS emerging technology has the potential to significantly enhance students’ perceived engagement, and the more they work on VS, the more they develop the values of attentive listening, directing attention, and investing effort in learning. When we eliminate the time factor and only compare different STEAM conditions, we can also conclude that STEAM technology-enhanced activities can contribute to the development of student-perceived engagement to a greater extent compared to non-technology ones. This contribution is significant in terms of behavioral engagement, which was achieved through VS integration within STEAM lessons. The order of integration of VS also improves perceived engagement, and students who learn with them first perceive all types of engagement better.

9.2 Contribution

Assessment of student engagement in education is of exceptional importance, especially for educators and practitioners, because it has been shown through various observations that it greatly affects all other teaching and learning outcomes of students, and that aspect can improve teaching performance and make it more personal and interesting to them. The modified EBCA scale can be used as a valid and reliable instrument for these purposes in working with primary school students;

Based on the assessment of student engagement with the use of the modified EBCA scale, teachers can adjust, dose, and adapt their teaching style, motivational support, and instructional guidance to the needs of students and thereby improve learning. In our study, it was shown that autonomy support, i.e., classroom conditions in which students feel free to express opinions, pursue interests, and ask questions, greatly influences the development of both agentic and all other types of engagement, which has the potential to transform and strengthen learning and bring it closer to students;

In addition to the above, the use of this scale in the assessment of student engagement can show teachers how students emotionally, behaviorally, cognitively, and agentically experience teaching activities, i.e., how they react, how they behave, how they learn, and what they undertake within the teaching process, which can direct them and help them in further adequately designing STEAM lessons according to the needs and interests of the children. Our study offers clear insights into this, as well as an example of a STEAM activity that can support teaching practice from this aspect;

In previous studies, it was confirmed that teachers who work with students of lower school age focus more on activating the behavioral components of engagement, while teachers who work with higher school students focus more on activating the cognitive components of engagement (Birch & Ladd, 1997 ; Greene et al., 2004 ; Reeve & Tseng, 2011 ), which did not prove to be the best in teaching practice. Assessment of student engagement using the EBCA scale can help teachers focus on redesigning teaching activities, i.e., on balancing and equally activating all types of student engagement, because in this way all components important for the learning process and students themselves can be ensured. The results of our study confirm that.

9.3 Implicationas for future studies and limitations

Given that our study is limited in terms of the generalization of the results because the model fit of the engagement scale was checked on a smaller sample of students aged 9–10 from Eastern Europe, the modified EBCA scale could be used for the same purposes in work with students of different primary grade levels and from different ethnic and cultural backgrounds, which will improve the generalization of the results and affect their applicability on a more global level;

Within our study, only one variable was tested: student engagement and it is recommended that its number be expanded (for example, variables achievement and motivation could be tested) and correlated with student engagement. In this sense, the modified EBCA scale can be used to assess whether and to what extent different types of engagement can predict student achievement and their motivation to learn. In this way, it is possible to discover which type of engagement predicts to the greatest extent student achievements and motivation, which is essential for teaching practice;

Given that in our study only VS was tested within STEAM activities, it is suggested to integrate and test other emerging technologies as well, from the perspective of student engagement. It is also recommended to investigate this issue more deeply through a longitudinal study, which would indicate the potential of emerging technologies in maintaining student engagement as well as in the development of different types of engagement considering the time frame and acquired experience in STEAM activities. Also, it is desirable to connect and discuss the results obtained in those studies from the perspectives of cognitive load theory and TAM theory and address the changes in education that STEAM enhanced with different emerging technologies can bring.

Code availability

(software application or custom code): Not applicable.

Data availability

(data transparency): All data and materials as well as software applications or custom code support published claims and comply with field standards. The data generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abla, C., & Fraumeni, B. R. (2019). Student engagement: Evidence-based strategies to boost academic and socialemotional results . McREL International.

Anđić, B., Ulbrich, E., Dana-Picard, T., Cvjetićanin, S., Petrović, F., Lavicza, Z., & Maričić, M. (2023). A Phenomenography Study of STEM teachers’ conceptions of using three-Dimensional modeling and Printing (3DMP) in teaching. Journal of Science Education and Technology , 32 (1), 45–60. https://doi.org/10.1007/s10956-022-10005-0 .

Article   Google Scholar  

Anđić, B., Lavicza, Z., Vučković, D., Maričić, M., Ulbrich, E., Cvjetićanin, S., & Petrović, F. (2023a). 3D printers as a learning tool for cooperative learning in an inclusive classroom. International Journal of Disability Development and Education . https://doi.org/10.1080/1034912X.2023.2223495 .

Anđić, B., Maričić, M., Weinhandl, R., Mumzu, F., Schmidthaler, E., & Lavicza, Z. (2024). Longitudinal study of metaphors changes in secondary School teachers’ beliefs about 3D modeling and Printing. Education and Information Technologies . https://doi.org/10.1007/s10639-023-12408-x .

Astin, A. (1984). Student involvement: A development theory for higher education. Journal of College Student Development , 40 , 518–529.

Google Scholar  

Axelson, R. D., & Flick, A. (2010). Defining student engagement. Change: The Magazine of Higher Learning , 43 (1), 38–43. https://doi.org/10.1080/00091383.2011.533096 .

Barlow, A., & Brown, S. (2020). Correlations between modes of student cognitive engagement and instructional practices in undergraduate STEM courses. International Journal of STEM Education . https://doi.org/10.1186/s40594-020-00214-7 .

Birch, S. H., & Ladd, G. W. (1997). The student–teacher relationship and children’s early school adjustment. Journal of School Psychology , 35 , 61–79. https://doi.org/10.1016/S0022-4405(96)00029-5 .

Boy, G. A. (2013). From STEM to STEAM: toward a human-centred education, creativity & learning thinking. In Proceedings of the 31st European conference on cognitive ergonomics (Vol. 3, pp. 1–7). https://doi.org/10.1145/2501907.2501934

Byrne, B. M. (2010). Structural equation modeling with Amos: Basic concepts, applications, and Programming (2nd ed.). Taylor and Francis Group.

Campbell, C., Speldewinde, C., Howitt, C., & MacDonald, A. (2018). STEM practice in the early years. Creative Education , 9 (01), 11. https://doi.org/10.4236/ce.2018.91002 .

Chen, C. H., & Chu, Y. R. (2024). VR-assisted inquiry-based learning to promote students’ science learning achievements, sense of presence, and global perspectives. Education and Information Technologies . https://doi.org/10.1007/s10639-024-12620-3 .

Chen, C. C., & Huang, P. H. (2020). The effects of STEAM-based mobile learning on learning achievement and cognitive load. Interactive Learning Environments . https://doi.org/10.1080/10494820.2020.1761838 .

Craig, P., Cooper, C., Gunnell, D., Haw, S., Lawson, K., Macintyre, S., Ogilvie, D., Petticrew, M., Reeves, B., Sutton, M., & Thompson, S. (2012). Using natural experiments to evaluate population health interventions: New Medical Research Council guidance. Journal of Epidemiology Community Health , 66 (12), 1182–1186. https://doi.org/10.1136/jech-2011-200375 .

Crowder, M. J., & Hand, D. J. (2017). Analysis of repeated measures . Routledge.

D’Angelo, C., Rutstein, D., Harris, C., Bernard, R., Borokhovski, E., & Haertel, G. (2014). Simulations for STEM learning: Systematic review and Meta-analysis . SRI International.

Falloon, G. (2019). Using simulations to teach young students science concepts: An experiential learning theoretical analysis. Computers & Education , 135 , 138–159. https://doi.org/10.1016/j.compedu.2019.03.001 .

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research , 18 (1), 39–50. https://doi.org/10.2307/3151312 .

Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research , 74 (1), 59–109. https://doi.org/10.3102/00346543074001059 .

Garcia-Martinez, J. A., Fuentes-Abeledo, E. J., & Rodriguez-Machado, E. R. (2021). Attitudes towards the use of ICT in Costa Rican University students: The influence of sex, academic performance, and training in technology. Sustainability , 13 . https://doi.org/10.3390/su13010282 .

Glancy, M. A. W. (2014). Examination of integrated STEM curricula as a means toward quality K-12 engineering education (Research to practice). ASEE annual Conference and Expositio. https://doi.org/10.18260/1-2-20446

Greene, B. A., Miller, R. B., Crowson, M., Duke, B. L., & Akey, K. L. (2004). Predicting high school students’ cognitive engagement and achievement: Contributions of classroom perceptions and motivation. Contemporary Educational Psychology , 29 , 462–482. https://doi.org/10.1016/j.cedpsych.2004.01.006 .

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate Data Analysis: A Global Perspective (7th ed).). Prentice Hall.

Hair, J. F., Sarstedt, M., Ringle, C. M., & Gudergan, S. P. (2017). Advanced issues in partial least squares structural equation modeling . Sage Publications Inc.

Hong, J. C., Ye, J. H., Ho, Y. J., & Ho, H. Y. (2020). Developing Inquiry and hands-on learning model to guide STEAM lesson planning for Kindergarten Children. Journal of Baltic Science Education , 19 , 908–922.

Hughes, S. B., Corrigan, W. M., Grove, D., Andersen, B. S., & Wong, T. J. (2022). Integrating arts with STEM and leading with STEAM to increase science learning with equity for emerging bilingual learners in the United States. International Journal of STEM Education, 9 . https://doi.org/10.1186/s40594-022-00375-7

Janković, A., Maričić, M., & Cvjetićanin, S. (2023). Comparing Science Success of Primary School students in the Gamified Learning Environment via Kahoot and Quizizz. Journal of Computers in Education . https://doi.org/10.1007/s40692-023-00266-y .

Jones, V. R. (2014). Teaching STEAM: 21st century skills. Child Technology Engineering, 18 (4), 11–13.

Kang, N. H. (2019). A review of the effect of integrated STEM or STEAM (science, technology, engineering, arts, and mathematics) education in South Korea. Asia-Pacific Science Education . https://doi.org/10.1186/s41029-019-0034-y .

Kahu, E. R., Stephens, C. V., Leach, L., & Zepke, N. (2015). Linking academic emotions and student engagement: Mature-aged distance students’ transition to university. Journal of Further and Higher Education, 39 (4), 481–497. https://doi.org/10.1080/0309877X.2014.895305

Katyara, P., Dahri, K. H., Muhiuddin, G., & Shabroz, S. (2023). Impact of Technology On Student’s Engagement in different dimensions: Cognitive, behavioral, reflective and social Engagement. Webology , 19 (3), 3451–3464.

Khamhaengpol, A., Sriprom, M., & Chuamchaitrakool, P. (2021). Development of STEAM activity on nanotechnology to determine basic science process skills and engineering design process for high school students. Thinking Skills and Creativity , 39 . https://doi.org/10.1016/j.tsc.2021.100796 .

Laut, J., Bartolini, T., & Porfiri, M. (2015). Bioinspiring an interest in STEM. IEEE Transactions on Education , 58 (1), 48–55. https://doi.org/10.1109/TE.2014.232453 .

Leavy, A. M., Dick, L., Meletiou-Mavrotheris, M., Paparistodemou, E., & Stylianou, E. (2023). The prevalence and use of emerging technologies in STEAM education: A systematic review of the literature. Journal of Computer Assisted Learning . https://doi.org/10.1111/jcal.12806 .

Lim, B. R. (2004). Challenges and issues in designing inquiry on the web. British Journal of Educational Technology , 35 (5), 627–643. https://doi.org/10.1111/j.0007-1013.2004.00419.x .

Maričić, M., Cvjetićanin, S., Adamov, J., Olić Ninković, S., & Anđić, B. (2022a). How do Direct and Indirect hands-on instructions strengthened by the self-explanation Effect promote learning? Evidence from Motion Content. Research in Science Education . https://doi.org/10.1007/s11165-022-10054-w .

Maričić, M., Cvjetićanin, S., Andevski, M., & Anđić, B. (2022b). Effects of Withholding Answers Coupled with Physical Manipulation on Students’ Learning of Magnetism-related Science Content. Research in Science and Technological Education . https://doi.org/10.1080/02635143.2022.2066648 .

Maričić, M., Cvjetićanin, S., Anđić, B., Marić, M., & Petojević, A. (2023). Using instructive simulations to teach young students simple Science concepts: Evidence from electricity content. Journal of Research on Technology in Education . https://doi.org/10.1080/15391523.2023.2196460 .

Meletiou-Mavrotheris, M. (2019). Augmented reality in STEAM education. In M. Peters, & R. Heraud (Eds.), Encyclopedia of educational innovation . Springer. https://doi.org/10.1007/978-981-13-2262-4_128-1 .

Moreno-Guerrero, A. J., Soler-Costa, R., Marín-Marín, J., & López-Belmonte, J. (2021). Flipped learning and good teaching practices in secondary education. Comunicar , 29 (68), 1–11. https://doi.org/10.3916/C68-2021-09 .

Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and selfregulated learning: A model and seven principles of good feedback practice. Studies in Higher Education , 31 (2), 199–218. http://doi.org/10. 1002/j.2168-9830.2010.tb01056.x.

Olympiou, G., Zacharias, Z., & de Jong, T. (2013). Making the invisible visible: Enhancing students’ conceptual understanding by introducing representations of abstract objects in a simulation. Instructional Science , 41 (3), 575–596. https://doi.org/10.1007/s11251-012-9245-2 .

Perignat, E. (2018). J. Katz-Buonincontro (Ed.), STEAM in practice and research: An integrative literature review. Thinking Skills and Creativity https://doi.org/10.1016/j.tsc.2018.10.002 .

Reeve, J., & Tseng, C. M. (2011). Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology , 36 (4), 257–267. https://doi.org/10.1016/j.cedpsych.2011.05.002 .

Ritoša, A., Danielsson, H., Sjöman, M., Almqvist, L., & Granlund, M. (2020). Assessing School Engagement – Adaptation and Validation of Engagement Versus Disaffection with Learning: Teacher report in the Swedish Educational Context. Frontiers in Education , 5 . https://doi.org/10.3389/feduc.2020.521972 .

Sanina, A., Kutergina, E., & Balashov, A. (2020). The Co-Creative approach to digital simulation games in social science education. Computers & Education , 149 . https://doi.org/10.1016/j.compedu.2020.103813 .

Sarı, U., Duygu, E., Şen, Ö. F., & Kırındı, T. (2020). The Effect of STEM Education on scientific process skills and STEM awareness in Simulation Based Inquiry Learning Environment. Journal of Turkish Science Education , 17 (3), 387–405. https://doi.org/10.36681/tused.2020.34 .

Scalise, K., Timms, M., Moorjani, A., Clark, L., Holtermann, K., & Irvin, P. S. (2011). Student learning in science simulations: Design features that promote learning gains. Journal of Research in Science Teaching , 48 (9), 1050–1078. https://doi.org/10.1002/tea.20437 .

Sosa, N. E., Salinas, J., & de Benito, B. (2017). Emerging technologies (ETs) in education: A systematic review of the literature published between 2006 and 2016. International Journal of Emerging Technologies in Learning , 12 (5). https://doi.org/10.3991/ijet.v12i05.6939 .

Tabachnick, B. G., & Fidell, L. S. (2007). Experimental designs using ANOVA (Vol. 724). Thomson/Brooks/Cole.

Techakosit, S., & Nilsook, P. (2018). The development of STEM literacy using the learning process of Scientific Imagineering through AR. International Journal of Emerging Technologies in Learning . https://doi.org/10.3991/ijet.v13i01.7664 .

Thisgaard, M., & Makransky, G. (2017). Virtual learning simulations in high school: Effects on cognitive and non-cognitive outcomes and implications on the development of STEM academic and career choice. Frontiers in Psychology , 8 . https://doi.org/10.3389/fpsyg.2017.00805 .

Wahyuningsih, S., Nurjanah, N. E., Rasmani, U. E. E., Hafdah, R., Pudyaningtyas, A. R., & Syamsuddin, M. M. (2020). STEAM learning in early childhood education: A literature review. International Journal of Pedagogy and Teacher Education , 4 (1), 33–44.

Wang, X., Yang, D., Wen, M., Koedinger, K., & Rosé, C. P. (2015). Investigating how student ’s cognitive behavior in MOOC discussion forums affect learning gains. International Conference on Educational Data Mining . https://files.eric.ed.gov/fulltext/ED560568.pdf .

Wen, C. T., Liu, C. C., Chang, H. Y., Chang, C. J., Chang, M. H., Chiang, F., Yang, S. H., & Hwang, C. W., F. K (2020). Students guided inquiry with simulation and its relation to school science achievement and scientific literacy. Computers & Education , 149. https://doi.org/10.1016/j.compedu.2020.103830 .

Zainuddin, Z., Shujahat, M., Haruna, H., & Wah Chu, S. K. (2020). The role of gamified e-quizzes on student learning and engagement: An interactive gamification solution for a formative assessment system. Computers & Education , 145. https://doi.org/10.1016/j.compedu.2019.103729 .

Zhang, M. (2014). Who are interested in online science simulations? Tracking a trend of digital divide in internet use. Computers & Education , 76 , 205–214. https://doi.org/10.1016/j.compedu.2014.04.001 .

Zinan, W., & Sai, G. T. B. (2017). Students’ perceptions of their ict-based College English Course in China: A Case Study. Teaching English with Technology , 17 (3), 53–76.

Download references

Acknowledgements

The authors thank to the Project that supported the realization of this research, as well as the students who participated in the research.

This research was supported by; by the TransEET Project, funded by the European Union HORIZON-WIDERA − 2021- ACCESS − 03 − 01. Project, licensed under [grant number 101078875], and by the Quality of the education system of Serbia in the European perspective Project, licensed under [grant number: 179010].

Open access funding provided by Johannes Kepler University Linz.

Author information

Authors and affiliations.

Faculty of Education in Sombor, Department of Sciences and Management in Education, University of Novi Sad, Podgorička 4, Sombor, 25000, Republic of Serbia

Mirjana Maričić

Linz School of Education, Department of STEM Education Science Park 5, Johannes Kepler University, Altenberger Straße 69, Linz, 4040, Austria

Zsolt Lavicza

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Zsolt Lavicza .

Ethics declarations

Ethical approval.

All procedures followed were in accordance with the ethical standards and principles of the conducting a research of the School of Education, University of Novi Sad and School of Education, Johannes Kepler University in Linz. An ethical approval was not mandatory (necessary for the type of collected data (i.e. anonymous questionnaire data) at the time when the study was conducted.

Consent for publication

All authors have read and approved the final version of the article.

Informed consent

For the realization of the research permission (consent) was sought from primary school principals, school pedagogues and psychologists, as well as teachers from selected classes, parents of children as well as the children themselves. Participation in the study was voluntary. The data was collected, saved, and analyzed anonymously.

Research involving human participants - rights

The study involved human participants ( N  = 84 3rd grade students, aged 9–10) who voluntarily chose to participate in the research. All research participants are guaranteed anonymity.

Conflict of interest

The authors declare that they have no conflict of interest to disclose.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Maričić, M., Lavicza, Z. Enhancing student engagement through emerging technology integration in STEAM learning environments. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12710-2

Download citation

Received : 02 December 2023

Accepted : 10 April 2024

Published : 24 May 2024

DOI : https://doi.org/10.1007/s10639-024-12710-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Emerging technologies
• Virtual simulations
• STEAM learning environment
• Find a journal
• Publish with us
• Track your research