• 2.2 Research Methods
  • Introduction
  • 1.1 What Is Sociology?
  • 1.2 The History of Sociology
  • 1.3 Theoretical Perspectives in Sociology
  • 1.4 Why Study Sociology?
  • Section Summary
  • Section Quiz
  • Short Answer
  • Further Research
  • 2.1 Approaches to Sociological Research
  • 2.3 Ethical Concerns
  • 3.1 What Is Culture?
  • 3.2 Elements of Culture
  • 3.3 High, Low, Pop, Sub, Counter-culture and Cultural Change
  • 3.4 Theoretical Perspectives on Culture
  • 4.1 Types of Societies
  • 4.2 Theoretical Perspectives on Society
  • 4.3 Social Constructions of Reality
  • 5.1 Theories of Self-Development
  • 5.2 Why Socialization Matters
  • 5.3 Agents of Socialization
  • 5.4 Socialization Across the Life Course
  • 6.1 Types of Groups
  • 6.2 Group Size and Structure
  • 6.3 Formal Organizations
  • 7.1 Deviance and Control
  • 7.2 Theoretical Perspectives on Deviance and Crime
  • 7.3 Crime and the Law
  • 8.1 Technology Today
  • 8.2 Media and Technology in Society
  • 8.3 Global Implications of Media and Technology
  • 8.4 Theoretical Perspectives on Media and Technology
  • 9.1 What Is Social Stratification?
  • 9.2 Social Stratification and Mobility in the United States
  • 9.3 Global Stratification and Inequality
  • 9.4 Theoretical Perspectives on Social Stratification
  • 10.1 Global Stratification and Classification
  • 10.2 Global Wealth and Poverty
  • 10.3 Theoretical Perspectives on Global Stratification
  • 11.1 Racial, Ethnic, and Minority Groups
  • 11.2 Theoretical Perspectives on Race and Ethnicity
  • 11.3 Prejudice, Discrimination, and Racism
  • 11.4 Intergroup Relationships
  • 11.5 Race and Ethnicity in the United States
  • 12.1 Sex, Gender, Identity, and Expression
  • 12.2 Gender and Gender Inequality
  • 12.3 Sexuality
  • 13.1 Who Are the Elderly? Aging in Society
  • 13.2 The Process of Aging
  • 13.3 Challenges Facing the Elderly
  • 13.4 Theoretical Perspectives on Aging
  • 14.1 What Is Marriage? What Is a Family?
  • 14.2 Variations in Family Life
  • 14.3 Challenges Families Face
  • 15.1 The Sociological Approach to Religion
  • 15.2 World Religions
  • 15.3 Religion in the United States
  • 16.1 Education around the World
  • 16.2 Theoretical Perspectives on Education
  • 16.3 Issues in Education
  • 17.1 Power and Authority
  • 17.2 Forms of Government
  • 17.3 Politics in the United States
  • 17.4 Theoretical Perspectives on Government and Power
  • Introduction to Work and the Economy
  • 18.1 Economic Systems
  • 18.2 Globalization and the Economy
  • 18.3 Work in the United States
  • 19.1 The Social Construction of Health
  • 19.2 Global Health
  • 19.3 Health in the United States
  • 19.4 Comparative Health and Medicine
  • 19.5 Theoretical Perspectives on Health and Medicine
  • 20.1 Demography and Population
  • 20.2 Urbanization
  • 20.3 The Environment and Society
  • Introduction to Social Movements and Social Change
  • 21.1 Collective Behavior
  • 21.2 Social Movements
  • 21.3 Social Change

Learning Objectives

By the end of this section, you should be able to:

  • Recall the 6 Steps of the Scientific Method
  • Differentiate between four kinds of research methods: surveys, field research, experiments, and secondary data analysis.
  • Explain the appropriateness of specific research approaches for specific topics.

Sociologists examine the social world, see a problem or interesting pattern, and set out to study it. They use research methods to design a study. Planning the research design is a key step in any sociological study. Sociologists generally choose from widely used methods of social investigation: primary source data collection such as survey, participant observation, ethnography, case study, unobtrusive observations, experiment, and secondary data analysis , or use of existing sources. Every research method comes with plusses and minuses, and the topic of study strongly influences which method or methods are put to use. When you are conducting research think about the best way to gather or obtain knowledge about your topic, think of yourself as an architect. An architect needs a blueprint to build a house, as a sociologist your blueprint is your research design including your data collection method.

When entering a particular social environment, a researcher must be careful. There are times to remain anonymous and times to be overt. There are times to conduct interviews and times to simply observe. Some participants need to be thoroughly informed; others should not know they are being observed. A researcher wouldn’t stroll into a crime-ridden neighborhood at midnight, calling out, “Any gang members around?”

Making sociologists’ presence invisible is not always realistic for other reasons. That option is not available to a researcher studying prison behaviors, early education, or the Ku Klux Klan. Researchers can’t just stroll into prisons, kindergarten classrooms, or Klan meetings and unobtrusively observe behaviors or attract attention. In situations like these, other methods are needed. Researchers choose methods that best suit their study topics, protect research participants or subjects, and that fit with their overall approaches to research.

As a research method, a survey collects data from subjects who respond to a series of questions about behaviors and opinions, often in the form of a questionnaire or an interview. The survey is one of the most widely used scientific research methods. The standard survey format allows individuals a level of anonymity in which they can express personal ideas.

At some point, most people in the United States respond to some type of survey. The 2020 U.S. Census is an excellent example of a large-scale survey intended to gather sociological data. Since 1790, United States has conducted a survey consisting of six questions to received demographical data pertaining to residents. The questions pertain to the demographics of the residents who live in the United States. Currently, the Census is received by residents in the United Stated and five territories and consists of 12 questions.

Not all surveys are considered sociological research, however, and many surveys people commonly encounter focus on identifying marketing needs and strategies rather than testing a hypothesis or contributing to social science knowledge. Questions such as, “How many hot dogs do you eat in a month?” or “Were the staff helpful?” are not usually designed as scientific research. The Nielsen Ratings determine the popularity of television programming through scientific market research. However, polls conducted by television programs such as American Idol or So You Think You Can Dance cannot be generalized, because they are administered to an unrepresentative population, a specific show’s audience. You might receive polls through your cell phones or emails, from grocery stores, restaurants, and retail stores. They often provide you incentives for completing the survey.

Sociologists conduct surveys under controlled conditions for specific purposes. Surveys gather different types of information from people. While surveys are not great at capturing the ways people really behave in social situations, they are a great method for discovering how people feel, think, and act—or at least how they say they feel, think, and act. Surveys can track preferences for presidential candidates or reported individual behaviors (such as sleeping, driving, or texting habits) or information such as employment status, income, and education levels.

A survey targets a specific population , people who are the focus of a study, such as college athletes, international students, or teenagers living with type 1 (juvenile-onset) diabetes. Most researchers choose to survey a small sector of the population, or a sample , a manageable number of subjects who represent a larger population. The success of a study depends on how well a population is represented by the sample. In a random sample , every person in a population has the same chance of being chosen for the study. As a result, a Gallup Poll, if conducted as a nationwide random sampling, should be able to provide an accurate estimate of public opinion whether it contacts 2,000 or 10,000 people.

After selecting subjects, the researcher develops a specific plan to ask questions and record responses. It is important to inform subjects of the nature and purpose of the survey up front. If they agree to participate, researchers thank subjects and offer them a chance to see the results of the study if they are interested. The researcher presents the subjects with an instrument, which is a means of gathering the information.

A common instrument is a questionnaire. Subjects often answer a series of closed-ended questions . The researcher might ask yes-or-no or multiple-choice questions, allowing subjects to choose possible responses to each question. This kind of questionnaire collects quantitative data —data in numerical form that can be counted and statistically analyzed. Just count up the number of “yes” and “no” responses or correct answers, and chart them into percentages.

Questionnaires can also ask more complex questions with more complex answers—beyond “yes,” “no,” or checkbox options. These types of inquiries use open-ended questions that require short essay responses. Participants willing to take the time to write those answers might convey personal religious beliefs, political views, goals, or morals. The answers are subjective and vary from person to person. How do you plan to use your college education?

Some topics that investigate internal thought processes are impossible to observe directly and are difficult to discuss honestly in a public forum. People are more likely to share honest answers if they can respond to questions anonymously. This type of personal explanation is qualitative data —conveyed through words. Qualitative information is harder to organize and tabulate. The researcher will end up with a wide range of responses, some of which may be surprising. The benefit of written opinions, though, is the wealth of in-depth material that they provide.

An interview is a one-on-one conversation between the researcher and the subject, and it is a way of conducting surveys on a topic. However, participants are free to respond as they wish, without being limited by predetermined choices. In the back-and-forth conversation of an interview, a researcher can ask for clarification, spend more time on a subtopic, or ask additional questions. In an interview, a subject will ideally feel free to open up and answer questions that are often complex. There are no right or wrong answers. The subject might not even know how to answer the questions honestly.

Questions such as “How does society’s view of alcohol consumption influence your decision whether or not to take your first sip of alcohol?” or “Did you feel that the divorce of your parents would put a social stigma on your family?” involve so many factors that the answers are difficult to categorize. A researcher needs to avoid steering or prompting the subject to respond in a specific way; otherwise, the results will prove to be unreliable. The researcher will also benefit from gaining a subject’s trust, from empathizing or commiserating with a subject, and from listening without judgment.

Surveys often collect both quantitative and qualitative data. For example, a researcher interviewing people who are incarcerated might receive quantitative data, such as demographics – race, age, sex, that can be analyzed statistically. For example, the researcher might discover that 20 percent of incarcerated people are above the age of 50. The researcher might also collect qualitative data, such as why people take advantage of educational opportunities during their sentence and other explanatory information.

The survey can be carried out online, over the phone, by mail, or face-to-face. When researchers collect data outside a laboratory, library, or workplace setting, they are conducting field research, which is our next topic.

Field Research

The work of sociology rarely happens in limited, confined spaces. Rather, sociologists go out into the world. They meet subjects where they live, work, and play. Field research refers to gathering primary data from a natural environment. To conduct field research, the sociologist must be willing to step into new environments and observe, participate, or experience those worlds. In field work, the sociologists, rather than the subjects, are the ones out of their element.

The researcher interacts with or observes people and gathers data along the way. The key point in field research is that it takes place in the subject’s natural environment, whether it’s a coffee shop or tribal village, a homeless shelter or the DMV, a hospital, airport, mall, or beach resort.

While field research often begins in a specific setting , the study’s purpose is to observe specific behaviors in that setting. Field work is optimal for observing how people think and behave. It seeks to understand why they behave that way. However, researchers may struggle to narrow down cause and effect when there are so many variables floating around in a natural environment. And while field research looks for correlation, its small sample size does not allow for establishing a causal relationship between two variables. Indeed, much of the data gathered in sociology do not identify a cause and effect but a correlation .

Sociology in the Real World

Beyoncé and lady gaga as sociological subjects.

Sociologists have studied Lady Gaga and Beyoncé and their impact on music, movies, social media, fan participation, and social equality. In their studies, researchers have used several research methods including secondary analysis, participant observation, and surveys from concert participants.

In their study, Click, Lee & Holiday (2013) interviewed 45 Lady Gaga fans who utilized social media to communicate with the artist. These fans viewed Lady Gaga as a mirror of themselves and a source of inspiration. Like her, they embrace not being a part of mainstream culture. Many of Lady Gaga’s fans are members of the LGBTQ community. They see the “song “Born This Way” as a rallying cry and answer her calls for “Paws Up” with a physical expression of solidarity—outstretched arms and fingers bent and curled to resemble monster claws.”

Sascha Buchanan (2019) made use of participant observation to study the relationship between two fan groups, that of Beyoncé and that of Rihanna. She observed award shows sponsored by iHeartRadio, MTV EMA, and BET that pit one group against another as they competed for Best Fan Army, Biggest Fans, and FANdemonium. Buchanan argues that the media thus sustains a myth of rivalry between the two most commercially successful Black women vocal artists.

Participant Observation

In 2000, a comic writer named Rodney Rothman wanted an insider’s view of white-collar work. He slipped into the sterile, high-rise offices of a New York “dot com” agency. Every day for two weeks, he pretended to work there. His main purpose was simply to see whether anyone would notice him or challenge his presence. No one did. The receptionist greeted him. The employees smiled and said good morning. Rothman was accepted as part of the team. He even went so far as to claim a desk, inform the receptionist of his whereabouts, and attend a meeting. He published an article about his experience in The New Yorker called “My Fake Job” (2000). Later, he was discredited for allegedly fabricating some details of the story and The New Yorker issued an apology. However, Rothman’s entertaining article still offered fascinating descriptions of the inside workings of a “dot com” company and exemplified the lengths to which a writer, or a sociologist, will go to uncover material.

Rothman had conducted a form of study called participant observation , in which researchers join people and participate in a group’s routine activities for the purpose of observing them within that context. This method lets researchers experience a specific aspect of social life. A researcher might go to great lengths to get a firsthand look into a trend, institution, or behavior. A researcher might work as a waitress in a diner, experience homelessness for several weeks, or ride along with police officers as they patrol their regular beat. Often, these researchers try to blend in seamlessly with the population they study, and they may not disclose their true identity or purpose if they feel it would compromise the results of their research.

At the beginning of a field study, researchers might have a question: “What really goes on in the kitchen of the most popular diner on campus?” or “What is it like to be homeless?” Participant observation is a useful method if the researcher wants to explore a certain environment from the inside.

Field researchers simply want to observe and learn. In such a setting, the researcher will be alert and open minded to whatever happens, recording all observations accurately. Soon, as patterns emerge, questions will become more specific, observations will lead to hypotheses, and hypotheses will guide the researcher in analyzing data and generating results.

In a study of small towns in the United States conducted by sociological researchers John S. Lynd and Helen Merrell Lynd, the team altered their purpose as they gathered data. They initially planned to focus their study on the role of religion in U.S. towns. As they gathered observations, they realized that the effect of industrialization and urbanization was the more relevant topic of this social group. The Lynds did not change their methods, but they revised the purpose of their study.

This shaped the structure of Middletown: A Study in Modern American Culture , their published results (Lynd & Lynd, 1929).

The Lynds were upfront about their mission. The townspeople of Muncie, Indiana, knew why the researchers were in their midst. But some sociologists prefer not to alert people to their presence. The main advantage of covert participant observation is that it allows the researcher access to authentic, natural behaviors of a group’s members. The challenge, however, is gaining access to a setting without disrupting the pattern of others’ behavior. Becoming an inside member of a group, organization, or subculture takes time and effort. Researchers must pretend to be something they are not. The process could involve role playing, making contacts, networking, or applying for a job.

Once inside a group, some researchers spend months or even years pretending to be one of the people they are observing. However, as observers, they cannot get too involved. They must keep their purpose in mind and apply the sociological perspective. That way, they illuminate social patterns that are often unrecognized. Because information gathered during participant observation is mostly qualitative, rather than quantitative, the end results are often descriptive or interpretive. The researcher might present findings in an article or book and describe what he or she witnessed and experienced.

This type of research is what journalist Barbara Ehrenreich conducted for her book Nickel and Dimed . One day over lunch with her editor, Ehrenreich mentioned an idea. How can people exist on minimum-wage work? How do low-income workers get by? she wondered. Someone should do a study . To her surprise, her editor responded, Why don’t you do it?

That’s how Ehrenreich found herself joining the ranks of the working class. For several months, she left her comfortable home and lived and worked among people who lacked, for the most part, higher education and marketable job skills. Undercover, she applied for and worked minimum wage jobs as a waitress, a cleaning woman, a nursing home aide, and a retail chain employee. During her participant observation, she used only her income from those jobs to pay for food, clothing, transportation, and shelter.

She discovered the obvious, that it’s almost impossible to get by on minimum wage work. She also experienced and observed attitudes many middle and upper-class people never think about. She witnessed firsthand the treatment of working class employees. She saw the extreme measures people take to make ends meet and to survive. She described fellow employees who held two or three jobs, worked seven days a week, lived in cars, could not pay to treat chronic health conditions, got randomly fired, submitted to drug tests, and moved in and out of homeless shelters. She brought aspects of that life to light, describing difficult working conditions and the poor treatment that low-wage workers suffer.

The book she wrote upon her return to her real life as a well-paid writer, has been widely read and used in many college classrooms.

Ethnography

Ethnography is the immersion of the researcher in the natural setting of an entire social community to observe and experience their everyday life and culture. The heart of an ethnographic study focuses on how subjects view their own social standing and how they understand themselves in relation to a social group.

An ethnographic study might observe, for example, a small U.S. fishing town, an Inuit community, a village in Thailand, a Buddhist monastery, a private boarding school, or an amusement park. These places all have borders. People live, work, study, or vacation within those borders. People are there for a certain reason and therefore behave in certain ways and respect certain cultural norms. An ethnographer would commit to spending a determined amount of time studying every aspect of the chosen place, taking in as much as possible.

A sociologist studying a tribe in the Amazon might watch the way villagers go about their daily lives and then write a paper about it. To observe a spiritual retreat center, an ethnographer might sign up for a retreat and attend as a guest for an extended stay, observe and record data, and collate the material into results.

Institutional Ethnography

Institutional ethnography is an extension of basic ethnographic research principles that focuses intentionally on everyday concrete social relationships. Developed by Canadian sociologist Dorothy E. Smith (1990), institutional ethnography is often considered a feminist-inspired approach to social analysis and primarily considers women’s experiences within male- dominated societies and power structures. Smith’s work is seen to challenge sociology’s exclusion of women, both academically and in the study of women’s lives (Fenstermaker, n.d.).

Historically, social science research tended to objectify women and ignore their experiences except as viewed from the male perspective. Modern feminists note that describing women, and other marginalized groups, as subordinates helps those in authority maintain their own dominant positions (Social Sciences and Humanities Research Council of Canada n.d.). Smith’s three major works explored what she called “the conceptual practices of power” and are still considered seminal works in feminist theory and ethnography (Fensternmaker n.d.).

Sociological Research

The making of middletown: a study in modern u.s. culture.

In 1924, a young married couple named Robert and Helen Lynd undertook an unprecedented ethnography: to apply sociological methods to the study of one U.S. city in order to discover what “ordinary” people in the United States did and believed. Choosing Muncie, Indiana (population about 30,000) as their subject, they moved to the small town and lived there for eighteen months.

Ethnographers had been examining other cultures for decades—groups considered minorities or outsiders—like gangs, immigrants, and the poor. But no one had studied the so-called average American.

Recording interviews and using surveys to gather data, the Lynds objectively described what they observed. Researching existing sources, they compared Muncie in 1890 to the Muncie they observed in 1924. Most Muncie adults, they found, had grown up on farms but now lived in homes inside the city. As a result, the Lynds focused their study on the impact of industrialization and urbanization.

They observed that Muncie was divided into business and working class groups. They defined business class as dealing with abstract concepts and symbols, while working class people used tools to create concrete objects. The two classes led different lives with different goals and hopes. However, the Lynds observed, mass production offered both classes the same amenities. Like wealthy families, the working class was now able to own radios, cars, washing machines, telephones, vacuum cleaners, and refrigerators. This was an emerging material reality of the 1920s.

As the Lynds worked, they divided their manuscript into six chapters: Getting a Living, Making a Home, Training the Young, Using Leisure, Engaging in Religious Practices, and Engaging in Community Activities.

When the study was completed, the Lynds encountered a big problem. The Rockefeller Foundation, which had commissioned the book, claimed it was useless and refused to publish it. The Lynds asked if they could seek a publisher themselves.

Middletown: A Study in Modern American Culture was not only published in 1929 but also became an instant bestseller, a status unheard of for a sociological study. The book sold out six printings in its first year of publication, and has never gone out of print (Caplow, Hicks, & Wattenberg. 2000).

Nothing like it had ever been done before. Middletown was reviewed on the front page of the New York Times. Readers in the 1920s and 1930s identified with the citizens of Muncie, Indiana, but they were equally fascinated by the sociological methods and the use of scientific data to define ordinary people in the United States. The book was proof that social data was important—and interesting—to the U.S. public.

Sometimes a researcher wants to study one specific person or event. A case study is an in-depth analysis of a single event, situation, or individual. To conduct a case study, a researcher examines existing sources like documents and archival records, conducts interviews, engages in direct observation and even participant observation, if possible.

Researchers might use this method to study a single case of a foster child, drug lord, cancer patient, criminal, or rape victim. However, a major criticism of the case study as a method is that while offering depth on a topic, it does not provide enough evidence to form a generalized conclusion. In other words, it is difficult to make universal claims based on just one person, since one person does not verify a pattern. This is why most sociologists do not use case studies as a primary research method.

However, case studies are useful when the single case is unique. In these instances, a single case study can contribute tremendous insight. For example, a feral child, also called “wild child,” is one who grows up isolated from human beings. Feral children grow up without social contact and language, which are elements crucial to a “civilized” child’s development. These children mimic the behaviors and movements of animals, and often invent their own language. There are only about one hundred cases of “feral children” in the world.

As you may imagine, a feral child is a subject of great interest to researchers. Feral children provide unique information about child development because they have grown up outside of the parameters of “normal” growth and nurturing. And since there are very few feral children, the case study is the most appropriate method for researchers to use in studying the subject.

At age three, a Ukranian girl named Oxana Malaya suffered severe parental neglect. She lived in a shed with dogs, and she ate raw meat and scraps. Five years later, a neighbor called authorities and reported seeing a girl who ran on all fours, barking. Officials brought Oxana into society, where she was cared for and taught some human behaviors, but she never became fully socialized. She has been designated as unable to support herself and now lives in a mental institution (Grice 2011). Case studies like this offer a way for sociologists to collect data that may not be obtained by any other method.

Experiments

You have probably tested some of your own personal social theories. “If I study at night and review in the morning, I’ll improve my retention skills.” Or, “If I stop drinking soda, I’ll feel better.” Cause and effect. If this, then that. When you test the theory, your results either prove or disprove your hypothesis.

One way researchers test social theories is by conducting an experiment , meaning they investigate relationships to test a hypothesis—a scientific approach.

There are two main types of experiments: lab-based experiments and natural or field experiments. In a lab setting, the research can be controlled so that more data can be recorded in a limited amount of time. In a natural or field- based experiment, the time it takes to gather the data cannot be controlled but the information might be considered more accurate since it was collected without interference or intervention by the researcher.

As a research method, either type of sociological experiment is useful for testing if-then statements: if a particular thing happens (cause), then another particular thing will result (effect). To set up a lab-based experiment, sociologists create artificial situations that allow them to manipulate variables.

Classically, the sociologist selects a set of people with similar characteristics, such as age, class, race, or education. Those people are divided into two groups. One is the experimental group and the other is the control group. The experimental group is exposed to the independent variable(s) and the control group is not. To test the benefits of tutoring, for example, the sociologist might provide tutoring to the experimental group of students but not to the control group. Then both groups would be tested for differences in performance to see if tutoring had an effect on the experimental group of students. As you can imagine, in a case like this, the researcher would not want to jeopardize the accomplishments of either group of students, so the setting would be somewhat artificial. The test would not be for a grade reflected on their permanent record of a student, for example.

And if a researcher told the students they would be observed as part of a study on measuring the effectiveness of tutoring, the students might not behave naturally. This is called the Hawthorne effect —which occurs when people change their behavior because they know they are being watched as part of a study. The Hawthorne effect is unavoidable in some research studies because sociologists have to make the purpose of the study known. Subjects must be aware that they are being observed, and a certain amount of artificiality may result (Sonnenfeld 1985).

A real-life example will help illustrate the process. In 1971, Frances Heussenstamm, a sociology professor at California State University at Los Angeles, had a theory about police prejudice. To test her theory, she conducted research. She chose fifteen students from three ethnic backgrounds: Black, White, and Hispanic. She chose students who routinely drove to and from campus along Los Angeles freeway routes, and who had had perfect driving records for longer than a year.

Next, she placed a Black Panther bumper sticker on each car. That sticker, a representation of a social value, was the independent variable. In the 1970s, the Black Panthers were a revolutionary group actively fighting racism. Heussenstamm asked the students to follow their normal driving patterns. She wanted to see whether seeming support for the Black Panthers would change how these good drivers were treated by the police patrolling the highways. The dependent variable would be the number of traffic stops/citations.

The first arrest, for an incorrect lane change, was made two hours after the experiment began. One participant was pulled over three times in three days. He quit the study. After seventeen days, the fifteen drivers had collected a total of thirty-three traffic citations. The research was halted. The funding to pay traffic fines had run out, and so had the enthusiasm of the participants (Heussenstamm, 1971).

Secondary Data Analysis

While sociologists often engage in original research studies, they also contribute knowledge to the discipline through secondary data analysis . Secondary data does not result from firsthand research collected from primary sources, but are the already completed work of other researchers or data collected by an agency or organization. Sociologists might study works written by historians, economists, teachers, or early sociologists. They might search through periodicals, newspapers, or magazines, or organizational data from any period in history.

Using available information not only saves time and money but can also add depth to a study. Sociologists often interpret findings in a new way, a way that was not part of an author’s original purpose or intention. To study how women were encouraged to act and behave in the 1960s, for example, a researcher might watch movies, televisions shows, and situation comedies from that period. Or to research changes in behavior and attitudes due to the emergence of television in the late 1950s and early 1960s, a sociologist would rely on new interpretations of secondary data. Decades from now, researchers will most likely conduct similar studies on the advent of mobile phones, the Internet, or social media.

Social scientists also learn by analyzing the research of a variety of agencies. Governmental departments and global groups, like the U.S. Bureau of Labor Statistics or the World Health Organization (WHO), publish studies with findings that are useful to sociologists. A public statistic like the foreclosure rate might be useful for studying the effects of a recession. A racial demographic profile might be compared with data on education funding to examine the resources accessible by different groups.

One of the advantages of secondary data like old movies or WHO statistics is that it is nonreactive research (or unobtrusive research), meaning that it does not involve direct contact with subjects and will not alter or influence people’s behaviors. Unlike studies requiring direct contact with people, using previously published data does not require entering a population and the investment and risks inherent in that research process.

Using available data does have its challenges. Public records are not always easy to access. A researcher will need to do some legwork to track them down and gain access to records. To guide the search through a vast library of materials and avoid wasting time reading unrelated sources, sociologists employ content analysis , applying a systematic approach to record and value information gleaned from secondary data as they relate to the study at hand.

Also, in some cases, there is no way to verify the accuracy of existing data. It is easy to count how many drunk drivers, for example, are pulled over by the police. But how many are not? While it’s possible to discover the percentage of teenage students who drop out of high school, it might be more challenging to determine the number who return to school or get their GED later.

Another problem arises when data are unavailable in the exact form needed or do not survey the topic from the precise angle the researcher seeks. For example, the average salaries paid to professors at a public school is public record. But these figures do not necessarily reveal how long it took each professor to reach the salary range, what their educational backgrounds are, or how long they’ve been teaching.

When conducting content analysis, it is important to consider the date of publication of an existing source and to take into account attitudes and common cultural ideals that may have influenced the research. For example, when Robert S. Lynd and Helen Merrell Lynd gathered research in the 1920s, attitudes and cultural norms were vastly different then than they are now. Beliefs about gender roles, race, education, and work have changed significantly since then. At the time, the study’s purpose was to reveal insights about small U.S. communities. Today, it is an illustration of 1920s attitudes and values.

As an Amazon Associate we earn from qualifying purchases.

This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax's permission.

Want to cite, share, or modify this book? This book uses the Creative Commons Attribution License and you must attribute OpenStax.

Access for free at https://openstax.org/books/introduction-sociology-3e/pages/1-introduction
  • Authors: Tonja R. Conerly, Kathleen Holmes, Asha Lal Tamang
  • Publisher/website: OpenStax
  • Book title: Introduction to Sociology 3e
  • Publication date: Jun 3, 2021
  • Location: Houston, Texas
  • Book URL: https://openstax.org/books/introduction-sociology-3e/pages/1-introduction
  • Section URL: https://openstax.org/books/introduction-sociology-3e/pages/2-2-research-methods

© Jan 18, 2024 OpenStax. Textbook content produced by OpenStax is licensed under a Creative Commons Attribution License . The OpenStax name, OpenStax logo, OpenStax book covers, OpenStax CNX name, and OpenStax CNX logo are not subject to the Creative Commons license and may not be reproduced without the prior and express written consent of Rice University.

Book cover

Principles of Social Research Methodology pp 313–321 Cite as

  • R. M. Channaveer 4 &
  • Rajendra Baikady 5  
  • First Online: 27 October 2022

1835 Accesses

1 Citations

This chapter reviews the strengths and limitations of case study as a research method in social sciences. It provides an account of an evidence base to justify why a case study is best suitable for some research questions and why not for some other research questions. Case study designing around the research context, defining the structure and modality, conducting the study, collecting the data through triangulation mode, analysing the data, and interpreting the data and theory building at the end give a holistic view of it. In addition, the chapter also focuses on the types of case study and when and where to use case study as a research method in social science research.

  • Qualitative research approach
  • Social work research

This is a preview of subscription content, log in via an institution .

Buying options

  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
  • Available as EPUB and PDF
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
  • Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Ang, C. S., Lee, K. F., & Dipolog-Ubanan, G. F. (2019). Determinants of first-year student identity and satisfaction in higher education: A quantitative case study. SAGE Open, 9 (2), 215824401984668. https://doi.org/10.1177/2158244019846689

Baxter, P., & Jack, S. (2015). Qualitative case study methodology: Study design and implementation for novice researchers. The Qualitative Report . Published. https://doi.org/10.46743/2160-3715/2008.1573

Bhatta, T. P. (2018). Case study research, philosophical position and theory building: A methodological discussion. Dhaulagiri Journal of Sociology and Anthropology, 12 , 72–79. https://doi.org/10.3126/dsaj.v12i0.22182

Article   Google Scholar  

Bromley, P. D. (1990). Academic contributions to psychological counselling. A philosophy of science for the study of individual cases. Counselling Psychology Quarterly , 3 (3), 299–307.

Google Scholar  

Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., & Sheikh, A. (2011). The case study approach. BMC Medical Research Methodology, 11 (1), 1–9.

Grässel, E., & Schirmer, B. (2006). The use of volunteers to support family carers of dementia patients: Results of a prospective longitudinal study investigating expectations towards and experience with training and professional support. Zeitschrift Fur Gerontologie Und Geriatrie, 39 (3), 217–226.

Greenwood, D., & Lowenthal, D. (2005). Case study as a means of researching social work and improving practitioner education. Journal of Social Work Practice, 19 (2), 181–193. https://doi.org/10.1080/02650530500144782

Gülseçen, S., & Kubat, A. (2006). Teaching ICT to teacher candidates using PBL: A qualitative and quantitative evaluation. Journal of Educational Technology & Society, 9 (2), 96–106.

Gomm, R., Hammersley, M., & Foster, P. (2000). Case study and generalization. Case study method , 98–115.

Hamera, J., Denzin, N. K., & Lincoln, Y. S. (2011). Performance ethnography . SAGE.

Hayes, N. (2000). Doing psychological research (p. 133). Open University Press.

Harrison, H., Birks, M., Franklin, R., & Mills, J. (2017). Case study research: Foundations and methodological orientations. In Forum qualitative sozialforschung/forum: Qualitative social research (Vol. 18, No. 1).

Iwakabe, S., & Gazzola, N. (2009). From single-case studies to practice-based knowledge: Aggregating and synthesizing case studies. Psychotherapy Research, 19 (4–5), 601–611. https://doi.org/10.1080/10503300802688494

Johnson, M. P. (2006). Decision models for the location of community corrections centers. Environment and Planning b: Planning and Design, 33 (3), 393–412. https://doi.org/10.1068/b3125

Kaarbo, J., & Beasley, R. K. (1999). A practical guide to the comparative case study method in political psychology. Political Psychology, 20 (2), 369–391. https://doi.org/10.1111/0162-895x.00149

Lovell, G. I. (2006). Justice excused: The deployment of law in everyday political encounters. Law Society Review, 40 (2), 283–324. https://doi.org/10.1111/j.1540-5893.2006.00265.x

McDonough, S., & McDonough, S. (1997). Research methods as part of English language teacher education. English Language Teacher Education and Development, 3 (1), 84–96.

Meredith, J. (1998). Building operations management theory through case and field research. Journal of Operations Management, 16 (4), 441–454. https://doi.org/10.1016/s0272-6963(98)00023-0

Mills, A. J., Durepos, G., & Wiebe, E. (Eds.). (2009). Encyclopedia of case study research . Sage Publications.

Ochieng, P. A. (2009). An analysis of the strengths and limitation of qualitative and quantitative research paradigms. Problems of Education in the 21st Century , 13 , 13.

Page, E. B., Webb, E. J., Campell, D. T., Schwart, R. D., & Sechrest, L. (1966). Unobtrusive measures: Nonreactive research in the social sciences. American Educational Research Journal, 3 (4), 317. https://doi.org/10.2307/1162043

Rashid, Y., Rashid, A., Warraich, M. A., Sabir, S. S., & Waseem, A. (2019). Case study method: A step-by-step guide for business researchers. International Journal of Qualitative Methods, 18 , 160940691986242. https://doi.org/10.1177/1609406919862424

Ridder, H. G. (2017). The theory contribution of case study research designs. Business Research, 10 (2), 281–305. https://doi.org/10.1007/s40685-017-0045-z

Sadeghi Moghadam, M. R., Ghasemnia Arabi, N., & Khoshsima, G. (2021). A Review of case study method in operations management research. International Journal of Qualitative Methods, 20 , 160940692110100. https://doi.org/10.1177/16094069211010088

Sommer, B. B., & Sommer, R. (1997). A practical guide to behavioral research: Tools and techniques . Oxford University Press.

Stake, R. E. (2010). Qualitative research: Studying how things work .

Stake, R. E. (1995). The Art of Case Study Research . Sage Publications.

Stoecker, R. (1991). Evaluating and rethinking the case study. The Sociological Review, 39 (1), 88–112.

Suryani, A. (2013). Comparing case study and ethnography as qualitative research approaches .

Taylor, S., & Berridge, V. (2006). Medicinal plants and malaria: An historical case study of research at the London School of Hygiene and Tropical Medicine in the twentieth century. Transactions of the Royal Society of Tropical Medicine and Hygiene, 100 (8), 707–714. https://doi.org/10.1016/j.trstmh.2005.11.017

Tellis, W. (1997). Introduction to case study. The Qualitative Report . Published. https://doi.org/10.46743/2160-3715/1997.2024

Towne, L., & Shavelson, R. J. (2002). Scientific research in education . National Academy Press Publications Sales Office.

Widdowson, M. D. J. (2011). Case study research methodology. International Journal of Transactional Analysis Research, 2 (1), 25–34.

Yin, R. K. (2004). The case study anthology . Sage.

Yin, R. K. (2003). Design and methods. Case Study Research , 3 (9.2).

Yin, R. K. (1994). Case study research: Design and methods (2nd ed.). Sage Publishing.

Yin, R. (1984). Case study research: Design and methods . Sage Publications Beverly Hills.

Yin, R. (1993). Applications of case study research . Sage Publishing.

Zainal, Z. (2003). An investigation into the effects of discipline-specific knowledge, proficiency and genre on reading comprehension and strategies of Malaysia ESP Students. Unpublished Ph. D. Thesis. University of Reading , 1 (1).

Zeisel, J. (1984). Inquiry by design: Tools for environment-behaviour research (No. 5). CUP archive.

Download references

Author information

Authors and affiliations.

Department of Social Work, Central University of Karnataka, Kadaganchi, India

R. M. Channaveer

Department of Social Work, University of Johannesburg, Johannesburg, South Africa

Rajendra Baikady

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to R. M. Channaveer .

Editor information

Editors and affiliations.

Centre for Family and Child Studies, Research Institute of Humanities and Social Sciences, University of Sharjah, Sharjah, United Arab Emirates

M. Rezaul Islam

Department of Development Studies, University of Dhaka, Dhaka, Bangladesh

Niaz Ahmed Khan

Department of Social Work, School of Humanities, University of Johannesburg, Johannesburg, South Africa

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Cite this chapter.

Channaveer, R.M., Baikady, R. (2022). Case Study. In: Islam, M.R., Khan, N.A., Baikady, R. (eds) Principles of Social Research Methodology. Springer, Singapore. https://doi.org/10.1007/978-981-19-5441-2_21

Download citation

DOI : https://doi.org/10.1007/978-981-19-5441-2_21

Published : 27 October 2022

Publisher Name : Springer, Singapore

Print ISBN : 978-981-19-5219-7

Online ISBN : 978-981-19-5441-2

eBook Packages : Social Sciences

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publish with us

Policies and ethics

  • Find a journal
  • Track your research

Log in using your username and password

  • Search More Search for this keyword Advanced search
  • Latest content
  • Current issue
  • BMJ Journals More You are viewing from: Google Indexer

You are here

  • Online First
  • Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment
  • Article Text
  • Article info
  • Citation Tools
  • Rapid Responses
  • Article metrics

Download PDF

  • http://orcid.org/0000-0001-9531-4990 Bashar Hasan 1 , 2 ,
  • http://orcid.org/0000-0001-9225-1197 Samer Saadi 1 , 2 ,
  • Noora S Rajjoub 1 ,
  • Moustafa Hegazi 1 , 2 ,
  • Mohammad Al-Kordi 1 , 2 ,
  • Farah Fleti 1 , 2 ,
  • Magdoleen Farah 1 , 2 ,
  • Irbaz B Riaz 3 ,
  • Imon Banerjee 4 , 5 ,
  • http://orcid.org/0000-0002-9368-6149 Zhen Wang 1 , 6 ,
  • http://orcid.org/0000-0001-5502-5975 Mohammad Hassan Murad 1 , 2
  • 1 Kern Center for the Science of Healthcare Delivery , Mayo Clinic , Rochester , Minnesota , USA
  • 2 Public Health, Infectious Diseases and Occupational Medicine , Mayo Clinic , Rochester , Minnesota , USA
  • 3 Division of Hematology-Oncology Department of Medicine , Mayo Clinic , Rochester , Minnesota , USA
  • 4 Department of Radiology , Mayo Clinic Arizona , Scottsdale , Arizona , USA
  • 5 School of Computing and Augmented Intelligence , Arizona State University , Tempe , Arizona , USA
  • 6 Health Care Policy and Research , Mayo Clinic Minnesota , Rochester , Minnesota , USA
  • Correspondence to Dr Bashar Hasan, Mayo Clinic, Rochester, MN 55905, USA; Hasan.Bashar{at}mayo.edu

Large language models (LLMs) may facilitate and expedite systematic reviews, although the approach to integrate LLMs in the review process is unclear. This study evaluates GPT-4 agreement with human reviewers in assessing the risk of bias using the Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) tool and proposes a framework for integrating LLMs into systematic reviews. The case study demonstrated that raw per cent agreement was the highest for the ROBINS-I domain of ‘Classification of Intervention’. Kendall agreement coefficient was highest for the domains of ‘Participant Selection’, ‘Missing Data’ and ‘Measurement of Outcomes’, suggesting moderate agreement in these domains. Raw agreement about the overall risk of bias across domains was 61% (Kendall coefficient=0.35). The proposed framework for integrating LLMs into systematic reviews consists of four domains: rationale for LLM use, protocol (task definition, model selection, prompt engineering, data entry methods, human role and success metrics), execution (iterative revisions to the protocol) and reporting. We identify five basic task types relevant to systematic reviews: selection, extraction, judgement, analysis and narration. Considering the agreement level with a human reviewer in the case study, pairing artificial intelligence with an independent human reviewer remains required.

  • Evidence-Based Practice
  • Systematic Reviews as Topic

Data availability statement

Data are available upon reasonable request. Search strategy, selection process flowchart, prompts and boxes containing included SRs and studies are available in the appendix. Analysed datasheet is available upon request.

https://doi.org/10.1136/bmjebm-2023-112597

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

Risk of bias assessment in systematic reviews is a time-consuming task associated with inconsistency. Large language models’ (LLMs) utilisation in systematic reviews may be helpful but largely unexplored.

WHAT THIS STUDY ADDS

This study introduces a structured framework for integrating LLMs into systematic reviews with four domains: rationale, protocol, execution and reporting.

The framework defines five possible task types for LLMs in systematic reviews: selection, data extraction, judgement, analysis and narration.

A case study about using LLMs for risk of bias assessments using Risk Of Bias In Non-randomised Studies of Interventions demonstrates fair agreement between LLM and human reviewers.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

The proposed framework can serve as a blueprint for future systematic reviewers planning to integrate LLMs into their workflow.

The case study suggests the need to pair LLMs assessing the risk of bias with a human reviewer.

Introduction

Systematic reviews are the key initial step in decision-making in healthcare. However, they are costly, require a long time to complete and become outdated, especially in areas of rapidly evolving evidence. Semi-automating systematic reviews and transitioning to living systematic reviews using the best contemporary available evidence are key priority areas of current evidence synthesis. 1–4 Recent advances in artificial intelligence (AI) have ushered in a new era of possibilities in healthcare practice and medical research, 5–7 including evidence synthesis and living systematic reviews. 8 9 By learning from human data analysis patterns (supervision), AI technologies offer the ability to automate, accelerate and enhance the accuracy of a wide array of research tasks, from data collection to analysis and even interpretation. 10

A recent AI advancement, large language models (LLMs) such as Meta AI LLaMA2 and OpenAI’s GPT-4, 11 are considered foundational models pre-trained in a self-supervised manner by leveraging a tremendous amount of free text data. The pre-training process allows them to acquire generic knowledge, and afterward, they can be fine-tuned on downstream tasks. With increasing model size, larger training data sets and longer training time, LLMs evolve emergent abilities such as zero-shot and few-shot in-context learning generalisation and have demonstrated significant capabilities in understanding and generating human-like text and processing data with minimal supervision, which may lead to meaningful participation in a systematic review. 12 13

Risk of bias (RoB) assessment is a significant step in systematic reviews that requires time, introduces inconsistencies and may be amenable to using AI and LLMs. 14 In this exposition, we propose a framework for incorporating LLMs into systematic reviews and employ GPT-4 for RoB assessment in a case study using the Cochrane Collaboration’s Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) tool. 15 We chose the ROBINS-I tool for this case study because it is a modern tool that is quite detailed, relatively complicated, and requires a long time to apply, 16 which makes it an ideal candidate to explore whether models such as GPT-4 can improve its consistency and time requirements.

The reporting of this case study adheres to the guidelines of methodological research. 17

Search strategy and study identification

We searched Scopus to identify all systematic reviews (SRs) from the Cochrane Collaboration that cited the original publication of the ROBINS-I tool. 15 We limited our search to SRs conducted by Cochrane in the field of medicine that were fully published. All original non-randomised studies included in the identified SRs were included as long as the ROBINS-I tool was used for their RoB assessment in the SR.

Data entry into ChatGPT

We conducted several pilot tests to determine the most effective method of obtaining RoB assessments using ChatGPT (GPT-4). The initial approach involved directly uploading the study PDFs to GPT-4 via the Code Interpreter tool available to Plus users. However, the tool was unable to interpret the fragmented pieces of text from the PDFs. We then attempted to paste the full text of individual studies in the prompt, however, this was unsuccessful due to the current estimated 2500-word limit for GPT-4 prompts. Finally, we converted the PDF to a Word file and extracted only the Methods and Results sections from each study for RoB assessment because these are the sections on which human reviewers focus for RoB assessments. Prompts used to instruct ChatGPT are presented in the appendix. The processes of data entry and prompt development were done iteratively until data were appropriately uploaded and a sensical output was obtained (ie, these processes were not prespecified). Foreign-language studies were provided in their original language to GPT-4.

Statistical analysis

One reviewer extracted RoB judgements from each Cochrane SR and a second reviewer verified the extraction. We measured the agreement between Cochrane reviewers and GPT-4 comparing the ordinal judgements about RoB using raw per cent agreement, weighted Cohen’s kappa and Kendall’s τ for correlation. The magnitude of agreement based on values of a correlation or kappa coefficient was considered to be slight (0–0.20), fair (0.21–0.40), moderate (0.41–60), substantial (0.61–0.80) and almost perfect (0.81–1.0).

Analysis was conducted using R software package (R Core Team (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria URL https://www.R-project.org ).

Initial screening and inclusion

The initial search yielded 98 SRs, from which 36 provided full ROBINS-I assessment. After deduplicating studies that appeared in multiple SRs, we finalised our sample with 307 unique individual studies ( online supplemental figure; box 1 and box 2 ).

Supplemental material

Agreement between cochrane reviewers and gpt-4.

Agreement measures are summarised in table 1 for each ROBINS-I domain and for overall judgements. Raw per cent agreement was the highest for the domain of ‘Classification of Intervention’. Kendall agreement coefficient was highest for the domains of ‘Participant Selection’, ‘Missing Data’ and ‘Measurement of Outcomes’, suggesting moderate agreement in these domains. Kappa coefficient was low across all domains. Agreement about the overall RoB across domains was fair (61% raw agreement, Kendall coefficient 0.35).

  • View inline

Performance metrics

Framework for incorporating LLM’s in a systematic review

Figure 1 outlines the proposed framework for integrating LLMs into a systematic review workflow. The framework has four domains that relate to establishing a rationale, incorporating LLM in the protocol of the systematic review, execution and reporting.

  • Download figure
  • Open in new tab
  • Download powerpoint

Framework for incorporating large language models in systematic reviews. LLM, large language model; RoB, risk of bias; SR, systematic review.

The first step is to establish the rationale (ie, why LLMs are needed, and whether they are capable of doing this specific task). In the protocol, the LLM model should be described with its version and whether it was off the shelf or used via other tools, applications or interfaces. For example, code interpreters or AI agents can be used. An LLM agent, such as a generative pre-trained transformer (GPT) agent, is a specialised system designed to execute complex, multistep tasks and can adapt to new tools not included in the general model’s training data or recently published tools.

The prompts for LLM need to be iteratively tested and refined and described in the protocol to the extent possible, realising that it will not be possible to prespecify or anticipate every step. The method of data entry (copy/paste vs uploading a file) also needs to be tested and described in the protocol. Metrics of success depend on the task type that is assigned to LLM. We identify five basic task types: selection (eg, of included studies), extraction (eg, of study characteristics and outcomes), judgement (eg, RoB assessment), analysis (quantitative and qualitative) and narration/editing (eg, writing a manuscript, abstract or a lay person or executive summary). The metrics of success and the extent of human interaction and supervision should also be specified in the protocol.

The execution of LLM engagement will likely lead to changes in some of the approaches specified in the protocol, which should be explicitly mentioned as revisions to the protocol. Reporting is the last part of the framework and is vital. The items mentioned above, which are beyond the usual reporting requirements from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement and its extensions, should all be included in the manuscript. 18 19 Importantly, the AI model and interface used need to be explicitly reported along with a timestamp of when AI was used because the output may vary over time for the same input and prompts. The transparency in reporting and informing peer reviewers and journal editors about the details of using LLMs are critical for the credibility of the systematic review process and subsequent decisions made based on the evidence. The proposed framework is applied to the current case study in table 2 .

Applying the proposed framework to the case study

The current case study suggests an overall fair agreement between Cochrane reviewers and ChatGPT-4 in using ROBINS-I for assessing RoB in non-randomised studies of intervention. This work identifies several challenges for using general utility LLM models, such as handling file types, word token limits and the quality of prompt engineering. Nonetheless, our study provides an assessment of zero-shot performance and a rationale for training RoB-specific systematic review models. The proposed framework is just a starting point since this field is very dynamic.

The current study also provides insight into evaluating inter-rater agreement on ordinal variables. We found that the weighted kappa coefficient was low across all domains which likely reflects the skewed distribution of the ratings. Kappa accounts for agreement occurring by chance, while Kendall’s τ measures the strength and direction of the association between two ranked variables. A recent comparison of reliability coefficients for ordinal rating scales suggested that the differences between these measures can vary at different agreement levels. 20 Thus, using more than one measure is helpful to assess the robustness of results. While our findings suggest the potential of LLMs like GPT-4 to be used in systematic reviews, it is obvious that there is a certain rate of error and that duplication of RoB assessment is needed.

Some limitations of the case study should be mentioned. This study was feasible because of the availability of comprehensive systematic reviews from the Cochrane Collaboration that used the ROBINS-I tool and reported detailed judgements. While their RoB assessment is certainly not a reference standard and can be quite poor for some domains such as confounding, 21 the rigorous and multidomain evaluation conducted by pairs of independent reviewers in these reviews makes them a reasonable comparison for novel LLM application. It is possible also that some systematic reviews used ROBINS-I but did not cite its original paper and were not included in our sample. We also had to use ChatGPT to translate a few studies published in languages other than English, truncate text when it was too lengthy and convert files format, all may have affected RoB judgements.

Practical implications

Given its current capabilities, GPT-4 is arguably a very advanced text-analysing tool. A major advantage is its availability as a universal language model—one model that can perform any language-based extraction, retrieval or even reasoning-based tasks. However, this approach may not be suitable for application in every domain. Sensitive domains like medicine require precise use of language in a consistent manner. LLMs have displayed trends of inconsistency in performance—different output for the same input. LLMs have the propensity to generate favourable answers and to hallucinate. Hallucination is a major threat to the use of LLMs in research. In table 3 , we describe the phenomenon of artificial hallucinations in terms of definition, types and plausible causes. 22–24

The phenomenon of artificial hallucinations: definition, types and causes

Additional applications in systematic reviews can extend to other tasks such as aiding in screening studies, translating foreign-language studies in real-time, data extraction, meta-analysis and even generating decision aids or translational products. 25 However, a human reviewer remains needed as a duplicate independent reviewer.

This exploration of LLMs application in systematic reviews is a step toward integrating AI as a dynamic adjunct in research. The proposed framework, coupled with a case study on RoB assessment, underscores the potential of LLMs to facilitate research tasks. While GPT-4 is not without its limitations, its ability to assist in complex tasks under human supervision makes it a promising tool for assessing RoB in systematic reviews. Considering the agreement level with a human reviewer in the case study, pairing AI with an independent human reviewer remains required at present.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

  • Chu H , et al
  • Sipra QUAR ,
  • Naqvi SAA , et al
  • Ryu AJ , et al
  • Naqvi SAA ,
  • He H , et al
  • Kayaalp ME ,
  • Ollivier M , et al
  • Noorbakhsh-Sabet N ,
  • Zhang Y , et al
  • Ramkumar PN ,
  • Haeberle HS , et al
  • Kelly SE , et al
  • Feng Y , et al
  • van Dijk SHB ,
  • Brusse-Keizer MGJ ,
  • Bucsán CC , et al
  • Touvron H ,
  • Stone K , et al
  • Kolluri S ,
  • Liu R , et al
  • Jardim PSJ ,
  • Ames HM , et al
  • Sterne JA ,
  • Hernán MA ,
  • Reeves BC , et al
  • Jeyaraman MM ,
  • Rabbani R ,
  • Al-Yousif N , et al
  • Liberati A ,
  • Tetzlaff J , et al
  • de Raadt A ,
  • Warrens MJ ,
  • Bosker RJ , et al
  • Thirunavukarasu AJ ,
  • Elangovan K , et al
  • Alkaissi H ,
  • McFarlane SI
  • Blaizot A ,
  • Veettil SK ,
  • Saidoung P , et al

Twitter @BasharHasanMD, @M_Hassan_Murad

Contributors MHM and BH conceived this study. BH, SS, MH, MA-K, FF, MF, ZW, IBR, IB and NSR participated in data identification, extraction and analysis. MHM, SS, IBR and IB wrote the first draft. All authors critically revised the manuscript and approved the final version. BH is the guarantor.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

COMMENTS

  1. Case Study Methodology of Qualitative Research: Key Attributes and

    A case study is one of the most commonly used methodologies of social research. This article attempts to look into the various dimensions of a case study research strategy, the different epistemological strands which determine the particular case study type and approach adopted in the field, discusses the factors which can enhance the effectiveness of a case study research, and the debate ...

  2. 2.2 Research Methods

    Recall the 6 Steps of the Scientific Method. Differentiate between four kinds of research methods: surveys, field research, experiments, and secondary data analysis. Explain the appropriateness of specific research approaches for specific topics. Sociologists examine the social world, see a problem or interesting pattern, and set out to study ...

  3. Case Study Methodology of Qualitative Research: Key Attributes and

    The following key attributes of the case study methodology can be underlined. 1. Case study is a research strategy, and not just a method/technique/process of data collection. 2. A case study involves a detailed study of the concerned unit of analysis within its natural setting. A de-contextualised study has no relevance in a case study ...

  4. PDF Case Study Research

    978-1-107-18126-7 — Case Study Research 2nd Edition ... Research Principles and Practices Case Study Research: Principles and Practices provides a general understanding of the case study method as well as speci c tools for its successful implemen- ... political science, psychology, social work, and sociology. Topics include: a survey of case ...

  5. PDF Chapter 21 Case Study

    Stoecker defined a case study as an "intensive research in which interpretations are given based on observable concrete interconnections between actual properties and people within an actual concrete setting" (Stoecker, 1991). There have been consistent efforts made to redefine case study for better under-standing.

  6. PDF Case Study Research in the Social Sciences

    Case study research has an important role in many social sciences including sociology, anthropology, political science, education, organizational studies, psychology, and nursing. ... For example, those who deny that case study research is a method rely on a narrower conception of methods than those who describe case study research as such. For ...

  7. The Case Method in Social Research

    The earliest applications of the case. method to social research were the. historians' descriptive accounts of peoples and nations, followed later by detailed studies of smaller groups, factions, and individuals. Case studies of contem- porary groups and individuals were a later development.3 In the study of society with its complex factors ...

  8. PDF Using Case Studies in The Social Sciences

    One can discern three main contexts of use: the case study as pedagogical tool, as method of treatment or intervention, and as an epistemic strategy. Even though, as a matter of fact, contexts of use to some extent match scientific fields, they do so loosely, in the sense that a given field can be hospitable to multiple uses of case studies.

  9. PDF Sociology 955 ETHNOGRAPHIC CASE STUDIES

    The journal itself will not be graded, but two weeks before the course is over, you will turn in a 2-3 page essay, based on your journal entries, on how your thinking about your project has evolved over the course of the semester. (10% of grade) 2. Participation: presentation of article. Over the course of the semester, each student will take ...

  10. PDF Revisiting Case Study Method of Social Research Examining Its Cardinal

    II. Some Definitions Of Case Study Yin (2009:18) defines case study as an empirical enquiry that investigates a contemporary phenomenon within its real-life context, where the boundaries between the phenomenon and context are not clearly defined, and in which multiple sources of data collection are used. Case study is not a method of data ...

  11. The Case Study as a Research Method

    December 1942, issues of the REVIEW OF EDUCATIONAL use of the case study in research methodology, progress has. this field. First, the case study has been of increased value. of research in education, psychology, sociology, and anthropology; progress has been made in the technics of gathering and study data for research purposes; and third ...

  12. What Is a Case, and What Is a Case Study?

    Hervé Dumez, i3-CRG (UMR 9217 - CNRS -École polytechnique), Batiment Ensta, 828 boulevard des Maréchaux, 91762 Palaiseau Cedex, France Email: [email protected] Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique. Total views and downloads: 8090. Article usage tracking started in December 2016.

  13. (PDF) Case study as a research method

    Abstract. Although case study methods remain a controversial approach to data collection, they are widely recognised in many social science studies especially when in-depth explanations of a ...

  14. (PDF) The Case Study in Social Research: History, Methods and Applications

    PDF | On Sep 5, 2023, Barbara Sena published The Case Study in Social Research: History, Methods and Applications | Find, read and cite all the research you need on ResearchGate

  15. [PDF] A Typology for the Case Study in Social Science Following a

    The author proposes a typology for the case study following a definition wherein various layers of classificatory principle are disaggregated. First, a clear distinction is drawn between two parts: (1) the subject of the study, which is the case itself, and (2) the object, which is the analytical frame or theory through which the subject is viewed and which the subject explicates. Beyond this ...

  16. Case Study

    The definitions of case study evolved over a period of time. Case study is defined as "a systematic inquiry into an event or a set of related events which aims to describe and explain the phenomenon of interest" (Bromley, 1990).Stoecker defined a case study as an "intensive research in which interpretations are given based on observable concrete interconnections between actual properties ...

  17. (PDF) Case Study Method

    Unlike the case study that refers to a method, a case history refers to a record of an individual or even a group. Case histories are used in many disciplines such as psychology, sociology,

  18. PDF UNIT 14 CASE STUDY

    14.2.2 Characteristics. Some of the very important characteristics of the case study method are listed below: Case study is a deep, detailed and intensive study of a social unit; It is a method of qualitative analysis; It is a comprehensive study; In this study all the variables and traits are linked with one another;

  19. PDF The Extended Case Method*

    In order to illustrate and explicate the extended case method I return to a study con-ducted between 1968 and 1972 in the then newly independent African country of Zambia. Of all my studies I have chosen this one because it most effectively illustrates both the virtues and the limits of the extended case method. First, the virtues: the extended ...

  20. Case study research in the social sciences

    1. What is case study research? Case study research is variously referred to as a methodology, research design, method, research strategy, research approach, style of reasoning, and the like. 2 It is sometimes a matter of contention whether to label case study research in one way or another.

  21. (PDF) Research Methods and Key Issues in Sociological Research

    Sociology is defined as the systematic study of human society (Macionis, 2003). Sociologists conduct Sociologists conduct research on the problems, development and interactions o f human society .

  22. The Case Study Method in Social Inquiry (Robert Stake, 1978)

    The Case Study Method in Social Inquiry (Robert Stake, 1978) - Free download as PDF File (.pdf), Text File (.txt) or read online for free. Estudo sobre o método de estudo de caso

  23. Integrating large language models in systematic reviews: a framework

    Large language models (LLMs) may facilitate and expedite systematic reviews, although the approach to integrate LLMs in the review process is unclear. This study evaluates GPT-4 agreement with human reviewers in assessing the risk of bias using the Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) tool and proposes a framework for integrating LLMs into systematic reviews. The ...