research paper survey results

If the statistical significance or p-value for a data point is equal to or lower than 0.05, it has moderate statistical significance since the probability for error is less than 5%. If the p-value is lower than 0.01, that means it has high statistical significance because the probability for error is less than 1%.

6. Consider causation versus correlation.

Another important aspect of survey analysis is knowing whether the conclusions you're drawing are accurate. For instance, let's say we observed a correlation between ice cream sales and car thefts in Boston. Over a month, as ice cream sales increased so did reports of stolen cars. While this data may suggest a link between these variables, we know that there's probably no relationship.

Just because the two are correlated doesn't mean one causes the other. In cases like these, there's typically a third variable — the independent variable — that influences the two dependent variables. In this case, it's temperature. As the temperature increases, more people buy ice cream. Additionally, more people leave their homes and go out, which leads to more opportunities for crime.

While this is an extreme example, you never want to draw a conclusion that's inaccurate or insufficient. Analyze all the data before assuming what influences a customer to think, feel, or act a certain way.

7. Compare new data with past data.

While current data is good for keeping you updated, it should be compared to data you've collected in the past. If you know 33% of respondents said they would recommend your brand, is that better or worse than last year? How about last quarter?

If this is your first year analyzing data, make these results the benchmark for your next analysis. Compare future results to this record and track changes over quarters, months, years, or whatever interval you prefer. You can even track data for specific subgroups to see if their experiences improve with your initiatives.

Now that you've gathered and analyzed all of your data, the next step is to share it with coworkers, customers, and other stakeholders. However, presentation is key in helping others understand the insights you're trying to explain.

The next section will explain how to present your survey results and share important customer data with the rest of your organization.

1. Use a graph or chart.

Graphs and charts are visually appealing ways to share data. Not only are the colors and patterns easy on the eyes, but data is often easier to understand when shared through a visual medium. However, it's important to choose a graph that highlights your results in a relevant way.

2. Minimal Formal Annual Report

This Canva report template lets the data speak for itself. The minimal portrait layout offers plenty of negative space around the content so that it can breathe. Bold numbers and percentages can remain or be omitted depending on the needs you have for each page. One of the rare gems of this template is its ability to balance large, clear images that don't crowd out the important written information on the page. Use this template for hybrid text-visual designs.

4. Empowerment Keynote Presentation

This presentation template makes a great research report template due to its clean lines, contrasting graphic elements, and ample room for visuals. The headers in this template virtually jump off the page to grab the readers' attention. There's aren't many ways to present quantitative data using this template example, but it works well for qualitative survey reports like focus groups or product design studies where original images will be discussed.

Don't forget to share this post!

Related articles.

Nonresponse Bias: What to Avoid When Creating Surveys

How to Make a Survey with a QR Code

50 Catchy Referral Slogans & How to Write Your Own

How Automated Phone Surveys Work [+Tips and Examples]

Online Panels: What They Are & How to Use Them Effectively

The Complete Guide to Survey Logic (+Expert Tips)

Focus Group vs. Survey: Which One Should You Use?

Leading Questions: What They Are & Why They Matter [+ Examples]

What are Survey Sample Sizes & How to Find Your Sample Size

28 Questionnaire Examples, Questions, & Templates to Survey Your Clients

Free Guide & Templates to Help Your Market Research

Service Hub provides everything you need to delight and retain customers while supporting the success of your whole front office

We use essential cookies to make Venngage work. By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts.

Manage Cookies

Cookies and similar technologies collect certain information about how you’re using our website. Some of them are essential, and without them you wouldn’t be able to use Venngage. But others are optional, and you get to choose whether we use them or not.

Strictly Necessary Cookies

These cookies are always on, as they’re essential for making Venngage work, and making it safe. Without these cookies, services you’ve asked for can’t be provided.

Show cookie providers

• Google Login

Functionality Cookies

These cookies help us provide enhanced functionality and personalisation, and remember your settings. They may be set by us or by third party providers.

Performance Cookies

These cookies help us analyze how many people are using Venngage, where they come from and how they're using it. If you opt out of these cookies, we can’t get feedback to make Venngage better for you and all our users.

• Google Analytics

Targeting Cookies

These cookies are set by our advertising partners to track your activity and show you relevant Venngage ads on other sites as you browse the internet.

• Google Tag Manager
• Infographics
• Daily Infographics
• Popular Templates
• Accessibility
• Graphic Design
• Graphs and Charts
• Data Visualization
• Human Resources
• Beginner Guides

Learn to Communicate with Data

How to present survey results using infographics.

How can you present survey data in a way that won’t bore your audience to tears?

Well, we all know that unique visuals like infographics can make charts and graphs more engaging. Survey data is easily translated into graphs and charts, making survey results and infographics the perfect marriage!

So without further ado, let’s get into everything you need to know to make a survey results infographic .

First up, let's kick things off by checking out some survey results templates that match up with different types of data. After that, I'll guide you through creating eye-catching survey results infographics, spicing up your results with some handy tips.

CREATE A FREE SURVEY RESULTS INFOGRAPHIC  

Click to jump ahead:

How to present survey results

• 3 types of survey results infographics

5 best practices for presenting survey results in infographics

Visualizing survey data effectively means using different types of charts for different types of survey results (i.e. binary, rating scale, multiple choice, single choice, or demographic results).

1. Binary results

If your survey questions offer two binary options (for example, “yes” and “no”), a pie chart is the simplest go-to option.

Using pies for binary results is pretty self-explanatory. Basically, just use a single pie slice to highlight the proportion of “Yes” responses compared to “No” responses. For the “Yes” responses, use a brighter, more saturated color and start the segment at 12 o’clock on the pie chart:

EDIT THIS SURVEY RESULTS TEMPLATE   

If you want to compare the response rates of multiple groups, skip the pies and go for a single bar chart. A bunch of aligned bars are much easier to compare than multiple pie charts. Don’t forget to label each bar with its percentage for clarity:

For a fun alternative that’s less information-dense, you can split up the bars to make a sort of modified 100% stacked bar chart. This frees up some space to add better labels for both the “Yes” responses and the “No” responses.

Or, forget about the extra notes and let the data speak for itself. Use a standard 100% stacked bar chart, color-coded to contrast the different responses, and sorted for readability.

2. Rating scale results

In a rating scale question, survey takers are offered a spectrum of possible answers and are asked to select an answer along that spectrum.

This type of question is often found on customer satisfaction surveys , used to gain an understanding of customer sentiment about a product or service. It's also popular for post event surveys , to gage how much people enjoyed the event. 

Most commonly it comes in one of two forms: the Likert scale (“Strongly Disagree,” “Disagree,” Neutral, “Agree” and “Strongly Agree”) or the Net Promoter Score (NPS, ranging from 0 to 10). The NPS is used to judge the willingness of a customer to recommend a product or service to others.

The 100% stacked bar chart is the simplest option for visualizing survey data from rating scale questions. It’s quick to make, and presents the proportion of responses in each category quite clearly.

With either of these scales, it’s helpful to summarize the results into coarser categories. Take the five- and ten-point Likert and NPS scales and summarize them into simpler three-point scales (“disagree”, “neutral”, and “agree” or “positive”, “neutral”, and “negative”).

Presenting survey results in a simplified categories goes a long way in making the chart easier to read.

3. Demographic results

If your survey gathers information about the respondents’ demographics in addition to other survey results, you may want to use that data as part of your analysis. Including factors like age, gender, income level, and even geographic location can make for an interesting infographic.

Visualizing survey data on a map is a fun way to include a demographic component in your infographic. A chloropleth map, like you see below, can be used to show the distribution of some data by geographic location . Different values are represented by different shades of a given color, so no reading is required:

Histograms, on the the other hand, can be used to show the age distribution of a particular population. They can easily incorporate data on gender, too:

While these specialized survey charts are great for more complex data, they won’t always be necessary. Consider using an icon chart when you want to make a simpler type of demographic data, like job or role, a feature of your design. They’re a fun way to add more impact to simple results.

4. Open-ended comments

Open-ended questions (questions that require respondents to write out their own answer, rather than selecting a preset answer) present a bit of a challenge. In order to visualize them, the answers need to be grouped in some way, either through common keywords, sentiments or some other factor.

Word clouds, though frowned upon by some data visualization experts, can be a quick way to get summary of this type of qualitative data.

They’re great for audiences who don’t have experience with data-heavy tables or statistical analysis , and they’re easy to make. Just pick out the most frequently-used keywords from the comments and plug them into our word cloud generator.

Otherwise you’ll have to do a more intensive manual qualitative analysis. Go through the open-ended responses and create categories.

Once you’ve quantified your answers, you’ll be able to present the results in a bar chart like this one, which shows the percent of comments that fall into each category.

5. Multiple choice results

Multiple choice questions allow respondents to select one or more answers from a list of possible answers.

The best visual for this kind of survey is a simple bar chart.

For the questions that allow respondents to make more than one selection, you’ll need to calculate the percentage of people who chose each answer, like you see in this chart from CoSchedule :

As always, bars should be sorted from greatest to least.

Pie charts are a decent option for times when respondents can only select a single answer. Keep in mind, though, that they’re not ideal if you’ve got a lot of data. If you have more than a few different responses to show, try giving each one its own chart:

3 Types of survey results infographics

Now that we’ve covered the best chart types for each type of survey result, let’s get into how we might combine survey charts to make a complete infographic.

A survey results infographic should use a combination of charts, graphic elements, and annotations tell a story.

Single-column summary infographics

The most popular type of survey results infographic is the single-column summary infographic. It sums up all of the major takeaways of a survey, explicitly stating the most important insights.

It might show the results of every survey question simply, using a large, bold number or basic chart for each question:

Or it might present a comprehensive overview of the data, with a more detailed, annotated chart for each survey question:

It might add some extra commentary after each question, too.

Either way, it presents the questions sequentially, in a single column, so that viewers can scroll through to read the results like a story.

To make your own single-column summary infographic, simply start at the top with the first question, and work your way down until you’ve covered each of the major survey insights. State each question, add the results in the form of a chart, and add notes about any interesting learnings.

To add some visual organization to a single-column infographic, use different background colors to create distinctions between sections. Add colored blocks behind each question to divide up the content.

Like you can see in the Netflix survey above, alternating red and black background colors adds a pleasing sense of rhythm and makes the infographic easier to scan.

Letter-sized summary infographics

If your survey is only a few questions long, a big single-column infographic is probably overkill. It might be better to stick with a basic 8.5”x11” page, and make it all about the numbers.

Forget about adding lots of notes, comments, and annotations. Just state each question in the simplest possible terms (i.e., “Where users are located”), and use simple survey charts to sum up the results.

Make sure you organize the charts based on an underlying grid , or you might end up with a jumbled mess.

Or you can even forget about charts altogether, and present the key takeaways as simply as possible. Use big, bold numbers to make a statement:

Letter-sized feature infographics

The last go-to option for presenting survey results is the one-page feature infographic. It couldn’t be more simple. It breaks down the results of a single survey question, in a single chart, on a single page.

We like to call this the “power stat” infographic. It combines a very simple chart with some big, bold text for a high-impact result:

Even if you have the most interesting survey data ever, no one will give it a second look if your infographic is poorly designed. Keep these best practices in mind when you make your next survey results infographic.

Clearly label charts to provide context and prevent misinterpretation

Your readers should be able to understand your survey charts in only a few seconds’ glance. Don't go for double barrelled questions when it comes to creating them. And if you ask me, that makes chart labels the most important chart elements (after the data itself, of course).

Descriptive labels can be used to add context to the data--to spell out the conclusions and implications of the data in the chart. This extra text will help to ensure that nothing is misinterpreted or lost in translation between you and your audience.

A well-labelled chart looks something like this:

The labels stand out against the background of the chart, with arrows clearly tying them to their respective data points.

Simplify the data to create clarity

It can be tempting to include every single data point in a visualization, but that won’t do you any good!

Be selective with your data. Just because you have a lot of data doesn’t mean your audience will want to spend hours scrolling through a mile-long infographic.

Select the most important results, and leave the rest for more in-depth summaries like white papers or reports . Include some supporting data if you need to, but remember--data visualization is all about cutting through the clutter .

Don’t embellish your infographic with unnecessary decorations

Along the same lines, avoid adding unnecessary icons, hard-to-read fonts, gaudy colors, 3D effects, or any other forms of “chartjunk”--ornamental elements that don’t help clarify anything about the data itself.

While you might think that adding extra elements will make your infographic more appealing,  they often only distract from the information you want to communicate.

The focus of your infographic should be A) the charts and B) your notes, labels, and annotations.

Apply style choices uniformly throughout the infographic

Regardless of what colors, fonts, images, or icons you use, be sure to apply styling consistently throughout the graphic.

Notice how color is used consistently (to represent the same response) in each section of this infographic?

That makes comparing responses across populations painless.

Include links to data sources in the infographic footer

Cite your data sources, ideally in link form, in the footer of your infographic. Make it easy for the more curious members of your audience to find and peruse the original data for themselves.

Even if it’s your own original research, linking to the complete data will help your credibility and allow readers to make their own decisions about the data. And who knows--maybe they’ll find something interesting that you missed the first time around!

Sometimes tables and graphs alone just don’t cut it.

While an in-depth analysis of survey results is best presented in a comprehensive report, an infographic is an excellent medium for summarizing your findings for more immediate impact.

Now that you know how to present survey results with the right charts, the infographic design process should be painless. If you get stuck, check out this roundup of our most popular survey results templates .

Or get started right away:

GET STARTED FOR FREE  

• Survey analysis
• Survey best practices
• Tips & tricks
• 5 Examples of How to Present Survey ...

5 Examples of How to Present Survey Results to Stakeholders

When you’ve lovingly designed, built, and distributed your survey and responses start flooding in, it’s time to begin the process of sorting and analyzing the data you’ll be presenting to stakeholders.

Once you’ve weeded the unusable responses, begin recording relevant responses through your survey platform or in a spreadsheet. If you use survey software like CheckMarket , you can easily transfer data into visuals with pre-built reports and dashboards.

Decide your data groups. Was the survey just answering one over-arching question? Or do you have multiple areas covered? Represent each data group separately.

For each result, provide additional information such as why you conducted the survey, what questions you were trying to answer, how the results help businesses, and any surprising answers.

When you have the data separated, the next step is to identify and prioritize the information your stakeholders will most want to see.

Choosing the Right Data to Share

First things first: who is your audience? Is it your boss? Is it your peers? Is it your direct clients or customers? The information that clients want to see, for instance, may be completely different to what your boss is interested in. The information you choose to share will vary drastically depending on the campaign you’re working on.

For example, if you’re working on a new marketing campaign, your audience may be interested in how you plan on advertising your business and what perks that may bring them.

However, when it comes to your stakeholders, they will be less interested in the customer perks, and more interested in how this new campaign will work for the business. They might want to know:

• How is it going to grow your audience?
• How will it turn them from leads to paying customers?
• How can this help improve your business’s bottom line?

When you’re presenting results, clearly define the purpose of the survey and why it matters to your stakeholders. Your story should be specific and concise.

Raise vital questions early on and have the answers ready to go. Your stakeholders have a limited amount of time to listen to what you have to say – make sure you are making the most of it.

This means you’ll have to pick and choose your data results carefully. All results need to be relevant and essential. Your stakeholders will be interested in information that makes a difference. And you’ll want the answers to be presented in the easiest way possible – which is why you want to choose your display method carefully.

5 Ways to Display Your Survey Results

When you present results, you are looking to be clear, simple, and memorable. So, viewers should not have to ask you to explain your results.

Here are five common ways to present your survey results to businesses, stakeholders, and customers.

1. Graphs and Charts

Graphs and charts summarize survey results in a quick, easy graphic for people to understand. Some of the most common types of graphs include:

• Bar graphs are the most popular way to display results. Easily create, customize, and show results. Most people also know how to read a basic bar graph to interpret survey results.
• Line graphs show how results change over time by tracking the ups and downs of the data.
• Pie charts show the breakup of a whole into sections. For example, your whole could be the total number of respondents, and the sections represent percentages that answered a certain way.
• Venn diagrams show the interaction between respondents and their answers. For example, overlapping circles could show the differences and similarities in responses between parents who use a product versus non-parents who use a product.

When creating a chart or graph, make the findings clear to read. Avoid too many intersecting lines and text options. If you can’t fit all the information into one graph, create several graphs rather than making one complex chart. Using colors to differentiate groups is another way to make results easy to read.

2. Infographics

Infographics add a creative twist to otherwise bland charts and graphs. A good infographic will use images to enhance the message, not distract from the data.

One survey results presentation example is to use silhouettes of people to convey a percentage of the population instead of a bar graph. This image helps those who see it connect the statistic to real people.

A word cloud is a powerful way to display open-ended question responses graphically. As more people respond with a specific word, that word will appear in the cloud – emphasizing the most relevant answers.

3. Video and Animations

People spend over 100 minutes a day watching videos – which is why marketers have tapped into this strategic area for reaching an audience. Nearly 88% of marketers say video marketing yields a strong return.

A video is a powerful tool for presenting information, including the results of your survey. You can capture your audience’s attention with motion, sound, and colorful statistics to help them remember information and react accordingly.

If you present findings through video, be aware that sharing options will be limited to platforms that can play a video – such a blog posts, websites, and PowerPoint presentations. Also, creating a PDF of the findings for people to look over at their leisure is a helpful way to support a video presentation.

4. Spreadsheets

Spreadsheets like Excel are not visually appealing, but they work well for organizing large amounts of information to create a survey results report.

While an image or video works best on websites, sometimes you may need to add more information than can fit in one picture.

Suppose you wanted to provide stakeholders or business partners with a detailed look at the survey and all the responses. A spreadsheet will allow the freedom to display all the necessary information at once. You can still use attractive infographics to summarize the findings and a video to present the report along with the spreadsheet.

5. Interactive Clickable Results

Interactive results are a fun way to allow viewers to explore results. You can also organize the findings to help break up large amounts of information.

Interactive maps are a common way to display survey results graphically. For example, results can be viewed by region when they click on a specific map area. Interactive maps and displays work best for websites and blogs.

An infographic that summarizes all the data as a global average allows people who don’t have the time to explore the map to see the information.

Customize Your Results in One Place

Time is precious in the marketing industry. You don’t want to spend days analyzing and sorting through survey results.

And you don’t have to.

By using CheckMarket, you can create, gather, and present survey results with one easy-to-use platform.

• Survey best practices (63)
• Market Research (62)
• Tips & tricks (52)
• Product updates (40)
• Company news (22)
• Customer Experience (19)
• Net Promoter Score (16)
• Employee Experience (16)
• Survey analysis (9)

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

• Skip to Guides Search
• Skip to breadcrumb
• Skip to main content
• Skip to footer
• Skip to chat link
• Report accessibility issues and get help
• Go to Penn Libraries Home
• Go to Franklin catalog

CWP: Craft of Prose: Researching the White Paper

• Getting started
• News and Opinion Sites
• Academic Sources
• Grey Literature
• Substantive News Sources
• What to Do When You Are Stuck
• Understanding a citation
• Examples of Quotation
• Examples of Paraphrase
• Chicago Manual of Style: Citing Images
• Researching the Op-Ed
• Researching Prospective Employers
• Resume Resources
• Cover Letter Resources

Research the White Paper

Researching the white paper:.

The process of researching and composing a white paper shares some similarities with the kind of research and writing one does for a high school or college research paper. What’s important for writers of white papers to grasp, however, is how much this genre differs from a research paper.  First, the author of a white paper already recognizes that there is a problem to be solved, a decision to be made, and the job of the author is to provide readers with substantive information to help them make some kind of decision--which may include a decision to do more research because major gaps remain. 

Thus, a white paper author would not “brainstorm” a topic. Instead, the white paper author would get busy figuring out how the problem is defined by those who are experiencing it as a problem. Typically that research begins in popular culture--social media, surveys, interviews, newspapers. Once the author has a handle on how the problem is being defined and experienced, its history and its impact, what people in the trenches believe might be the best or worst ways of addressing it, the author then will turn to academic scholarship as well as “grey” literature (more about that later).  Unlike a school research paper, the author does not set out to argue for or against a particular position, and then devote the majority of effort to finding sources to support the selected position.  Instead, the author sets out in good faith to do as much fact-finding as possible, and thus research is likely to present multiple, conflicting, and overlapping perspectives. When people research out of a genuine desire to understand and solve a problem, they listen to every source that may offer helpful information. They will thus have to do much more analysis, synthesis, and sorting of that information, which will often not fall neatly into a “pro” or “con” camp:  Solution A may, for example, solve one part of the problem but exacerbate another part of the problem. Solution C may sound like what everyone wants, but what if it’s built on a set of data that have been criticized by another reliable source?  And so it goes. 

For example, if you are trying to write a white paper on the opioid crisis, you may focus on the value of  providing free, sterilized needles--which do indeed reduce disease, and also provide an opportunity for the health care provider distributing them to offer addiction treatment to the user. However, the free needles are sometimes discarded on the ground, posing a danger to others; or they may be shared; or they may encourage more drug usage. All of those things can be true at once; a reader will want to know about all of these considerations in order to make an informed decision. That is the challenging job of the white paper author.     
 The research you do for your white paper will require that you identify a specific problem, seek popular culture sources to help define the problem, its history, its significance and impact for people affected by it.  You will then delve into academic and grey literature to learn about the way scholars and others with professional expertise answer these same questions. In this way, you will create creating a layered, complex portrait that provides readers with a substantive exploration useful for deliberating and decision-making. You will also likely need to find or create images, including tables, figures, illustrations or photographs, and you will document all of your sources. 

Latin American Studies Librarian

Connect to a Librarian Live Chat or "Ask a Question"

• Librarians staff live chat from 9-5 Monday through Friday . You can also text to chat: 215-543-7674
• You can submit a question 24 hours a day and we aim to respond within 24 hours 
• You can click the "Schedule Appointment" button above in librarian's profile box (to the left), to schedule a consultation with her in person or by video conference.  
• You can also make an appointment with a  Librarian by subject specialization . 
• Connect by email with a subject librarian

Find more easy contacts at our Quick Start Guide

• Next: Getting started >>
• Last Updated: Aug 26, 2024 1:21 PM
• URL: https://guides.library.upenn.edu/c.php?g=1419866

This paper is in the following e-collection/theme issue:

Published on 29.8.2024 in Vol 13 (2024)

Optimizing Response Rates to Examine Health IT Maturity and Nurse Practitioner Care Environments in US Nursing Homes: Mixed Mode Survey Recruitment Protocol

Authors of this article:

• Gregory L Alexander 1 * , RN, PhD   ; 
• Lusine Poghosyan 1 * , RN, MPH, PhD   ; 
• Yihong Zhao 1 * , PhD   ; 
• Mollie Hobensack 2 * , RN, PhD   ; 
• Sergey Kisselev 1 * , MA   ; 
• Allison A Norful 1 * , BSN, MSN, PhD, ANP-BC   ; 
• John McHugh 3 * , MBA, PhD   ; 
• Keely Wise 1 *   ; 
• M Brooke Schrimpf 1 * , BA   ; 
• Ann Kolanowski 4 * , RN, PhD   ; 
• Tamanna Bhatia 3 * , BS, BA   ; 
• Sabrina Tasnova 1 * , BA  

1 School of Nursing, Columbia University, New York, NY, United States

2 Icahn School of Medicine Mount Sinai, New York, NY, United States

3 School of Public Health, Columbia University Mailman, New York, NY, United States

4 Pennsylvania State University, University Park, PA, United States

*all authors contributed equally

Corresponding Author:

Gregory L Alexander, RN, PhD

School of Nursing

Columbia University

560 W. 168 Room 628

New York, NY, 10032

United States

Phone: 1 5733013131

Email: [email protected]

Background: Survey-driven research is a reliable method for large-scale data collection. Investigators incorporating mixed-mode survey designs report benefits for survey research including greater engagement, improved survey access, and higher response rate. Mix-mode survey designs combine 2 or more modes for data collection including web, phone, face-to-face, and mail. Types of mixed-mode survey designs include simultaneous (ie, concurrent), sequential, delayed concurrent, and adaptive. This paper describes a research protocol using mixed-mode survey designs to explore health IT (HIT) maturity and care environments reported by administrators and nurse practitioners (NPs), respectively, in US nursing homes (NHs).

Objective: The aim of this study is to describe a research protocol using mixed-mode survey designs in research using 2 survey tools to explore HIT maturity and NP care environments in US NHs.

Methods: We are conducting a national survey of 1400 NH administrators and NPs. Two data sets (ie, Care Compare and IQVIA) were used to identify eligible facilities at random. The protocol incorporates 2 surveys to explore how HIT maturity (survey 1 collected by administrators) impacts care environments where NPs work (survey 2 collected by NPs). Higher HIT maturity collected by administrators indicates greater IT capabilities, use, and integration in resident care, clinical support, and administrative activities. The NP care environment survey measures relationships, independent practice, resource availability, and visibility. The research team conducted 3 iterative focus groups, including 14 clinicians (NP and NH experts) and recruiters from 2 national survey teams experienced with these populations to achieve consensus on which mixed-mode designs to use. During focus groups we identified the pros and cons of using mixed-mode designs in these settings. We determined that 2 mixed-mode designs with regular follow-up calls (Delayed Concurrent Mode and Sequential Mode) is effective for recruiting NH administrators while a concurrent mixed-mode design is best to recruit NPs.

Results: Participant recruitment for the project began in June 2023. As of April 22, 2024, a total of 98 HIT maturity surveys and 81 NP surveys have been returned. Recruitment of NH administrators and NPs is anticipated through July 2025. About 71% of the HIT maturity surveys have been submitted using the electronic link and 23% were submitted after a QR code was sent to the administrator. Approximately 95% of the NP surveys were returned with electronic survey links.

Conclusions: Pros of mixed-mode designs for NH research identified by the team were that delayed concurrent, concurrent, and sequential mixed-mode methods of delivering surveys to potential participants save on recruitment time compared to single mode delivery methods. One disadvantage of single-mode strategies is decreased versatility and adaptability to different organizational capabilities (eg, access to email and firewalls), which could reduce response rates.

International Registered Report Identifier (IRRID): DERR1-10.2196/56170

Introduction

Survey use in clinical informatics research is ubiquitous. Surveys are often used to collect data and measure phenomena such as knowledge of clinical informatics specialties [ 1 ] or the use of electronic health records [ 2 ]. Benefits of using surveys include lower costs to conduct research, better population descriptions, flexibility, and dependability of study designs [ 3 ]. Surveys are used in many professions and across health care settings, including nursing homes, home health care, and hospitals [ 4 - 6 ]. The expansive use of surveys in clinical informatics research calls for a continued focus on training to improve the ability of researchers to design high-quality surveys, develop effective reporting mechanisms, maximize recruitment strategies, and adapt to recruitment challenges needed to enhance the results. Various modes of survey data collection exist across studies. Literature establishing a theoretical foundation for questionnaire response styles used in surveys when collecting data about public opinion indicate that mode of data collection (eg, mixed-modes) is an important stimulus for response [ 7 ]. In this paper, researchers describe a research protocol using mixed-mode survey designs in clinical informatics research using 2 survey tools to explore Health IT (HIT) maturity and nurse practitioner (NP) care environments in US nursing homes (NHs).

In this protocol, HIT maturity is defined in 3 dimensions including HIT capabilities, use, and integration. These HIT maturity dimensions are conceived within NH resident care, clinical support (eg, HIT use in laboratory, pharmacy, and radiology activities), and administrative activities [ 8 ]. The HIT maturity survey tool contains 27 content areas and 183 content items [ 9 ]. The tool will be used to survey NH administrators. The Nurse Practitioner Nursing Home Organizational Climate Questionnaire (NP-NHOCQ), used to measure NP care environments, contains 5 subscales and 41 items. This tool will be used to survey NPs in NHs. The NP-NHOCQ measures the care environment of NPs in NHs in 5 areas: (1) NP-Physician Relations, (2) NP-Administration Relations, (3) NP-Director of Nursing Relations, (4) Independent Practice and Support, and (5) Professional Visibility.

Mixed-Mode Survey Research

Survey-driven research is known as a reliable data collection method to capture individual perspectives on a large scale. However, there are many challenges related to survey-based data collection, such as low response rates and rising costs of human capital [ 10 ]. Previously, researchers have explored the use of mixed-mode survey designs combining methods such as web, phone, face-to-face, and mail administrations. Mixed-mode survey research involves using 2 or more of these modes for data collection [ 11 ]. A survey mode is defined as the communication channel used to collect survey data from one or more respondents [ 11 ]. Prior research has reported the benefits of mixed-mode surveys such as enhancing engagement [ 12 ], mitigating accessibility barriers [ 13 ], and increasing response rates [ 14 ].

Survey modes can be implemented individually or combined with other modes. A single mode approach deploys only one mode at a time. For example, a researcher may use postal mail services as the only method to contact study participants and collect data. Alternatively, mixed-mode designs use multiple modes to recruit respondents (see Figure 1 ). For instance, a simultaneous (also known as concurrent) mixed-mode approach allows respondents to choose their preference between multiple modes deployed at the same time. For instance, a researcher may offer study participants a choice to complete a survey using an electronic PDF version of the questionnaire that can be printed, scanned, and faxed back to researchers or an electronic survey link completed via web. Mixed-modes can also use a sequential approach. In this mode, researchers may offer 2 different modes, one mode at a time, with a second mode coming later, after the first. This mode is particularly useful when following up with participants who do not respond (nonrespondents) to provide alternative survey strategies that better suit their workflows. An example of sequential mode may include contacting participants initially via phone call and then, following no initial response, a second contact is made using a QR code that is sent via a mailed letter. Another mixed-mode useful for following up with nonrespondents is called a delayed concurrent mode. In this mode, participants are offered one mode, then nonrespondents are offered a choice between 2 other modes later during follow-up activities. An example of the delayed concurrent mode might include an initial mailed survey. Then when no response is received, potential participants are sent a choice between a face-to-face or a phone interview to complete the survey. Finally, an adaptive mixed-mode design incorporates different sampling units. In the adaptive modes, 2 different samples are each offered a different mode.

Mixed-mode survey research has long been identified as a means to improve participation in survey recruitment. For instance, a systematic review of 22 articles among nurses provided evidence that recruitment design strategies that include postal and telephone contacts are generally more successful than fax or web-based approaches [ 16 ]. In a more recent systematic review of 893 studies, mode of administration was a key factor in successful recruitment. However, in this review, electronic and postal modes of survey data collection were less likely to result in higher response rates [ 17 ]. In other research using mixed-modes with clinicians, using a multiple contact protocol generated final response rates 10% points higher than single mode methods [ 18 ].

In this paper, we present mixed-mode methods used in a large survey of NHs in the United States. To achieve research goals, we must have a robust and effective recruitment plan. Therefore, we are using an innovative research protocol using mixed-modes to improve NH administrators’ and NPs’ engagement in survey data collection while increasing the response rates.

Mixed-Mode Survey Research in Nursing Homes

The US health care system has over 15,600 NHs serving over 1.3 million residents [ 19 ]. A growing strategy for improving the outcomes for NH residents is to effectively integrate HIT into care delivery to promote safer care environments for NH residents. HIT integration into NH resident care may improve care environments and by extension, better care quality [ 20 ]. Survey-driven research is a reliable method to capture the perspective of individuals about these phenomena on a large scale. Our team is conducting a national survey of NH administrators and NPs, incorporating 2 different survey tools to explore how HIT maturity (survey 1) impacts care environments (survey 2) where NPs work. A specific aim of this research is to provide comprehensive assessments of HIT maturity and NP care environments in NHs nationally. The goal of the National Institute of Aging funded research study (5R01AG080517, principal investigators: GLA and LP) is to assess differences in HIT maturity and care environments in NHs where NPs deliver care to residents with Alzheimer disease and related dementias and examine their impact on hospitalizations and emergency department visits among residents.

The sample for this study includes randomly selected NHs including administrators (ie, NH leaders responsible for HIT systems in their organization) and NPs from each NH. Our goal is to recruit participants from 1400 NHs in the United States. We use 2 national sources to identify NHs for this study. The first data source is called NH Compare (or Care Compare), a publicly available national data set containing information about organizational characteristics of US NHs and quality of care [ 21 ]. The second data source stems from IQVIA, a company that stores national data about NH location, contact information, and staff including administrators and NPs. In preparation for this proposal, IQVIA provided our team data to identify all US NHs with practicing NPs. According to these data, in 2021, a total of 11,222 unique NPs worked in 5000 NHs for an average of 2.2 NPs/NH. Based on this estimate, we expect to contact 3080 NPs within the 1400 NHs (1400 NHs X 2.2 NPs/NH).

Inclusion and Exclusion Criteria

We use NH Compare files to identify NHs for our study based on 2 specific inclusion criteria. First, we include all NHs located in the United States including Alaska and Hawaii. Second, we include at least 1 NP working in each facility. NPs may include actual employees of a facility or may be employed by an external organization as a consultant for a facility and not directly by the NH. Facilities are not eligible to participate if they meet the following 3 exclusion criteria. First, NHs that do not have an NP employed. Second, NHs with a hospital-based designation as their HIT maturity are likely to be different due to national incentives for HIT adoption in acute care [ 22 , 23 ]. Approximately 6% (n=15,518) of NHs have a health system designation that includes common ownership or joint management [ 24 ]. Third, NHs that are designated as a special focus facility (SFF), which indicates any NH with a history of serious quality issues. NHs with an SFF designation are required to be in a program to stimulate quality-of-care improvements [ 25 ]. In October 2023, Centers for Medicare & Medicaid Services indicated that approximately 0.5% of US NHs have an SFF designation [ 25 ].

The NH Compare website was downloaded in February 2023 to identify facilities for recruitment. We identified 4163 facilities that matched our criteria. In preliminary work, during 2 prior NH survey studies, we achieved approximately a 45% response rate of surveys returned from administrators. Therefore, for the current protocol, we oversampled by randomly selecting 3000 NHs, which we identified by linking the NH Compare and IQVIA data. We included at least 5 facilities in each state, except for Alaska (2 facilities) and Wyoming (3 facilities) which have few NHs with NPs identified. We will recruit all administrators from these 3000 NHs to complete a HIT maturity survey. For every NH that completes the HIT maturity survey, we will recruit all NPs from those facilities.

Sample Characteristics

After we generated the random sample from the merged files, we compared basic characteristics of NHs between the selected NHs and the rest of the NHs nationally. The following NH characteristics were compared to assure that there was limited bias in sample representation:

• Bed size (<60 beds, 60-120 beds, and >120 beds)
• Ownership (for profit vs nonprofit)
• Location (metropolitan, micropolitan, small town, and rural)
• Staffing hour
• Medicare vs Medicaid
• NH overall rating: (ranging from 1 to 5)

Focus Groups to Assess the Pros and Cons of Mixed-Mode Designs

The research team conducted iterative focus groups that included NPs and survey recruitment experts to discuss the pros and cons of different recruitment strategies. To explore the pros and cons, members of the focus groups assessed recruitment strategies used during 2 prior national studies of long-term care NH sites [ 26 ]. The PI and some members of the focus groups lead these national studies that were reviewed. Additionally, members of the focus groups reviewed and discussed potential mixed-mode strategies from the literature to incorporate into this protocol. Schouten [ 15 ] mixed-mode survey research helped inform our decisions for our protocol design.

Data Collection

We aim to survey administrators and NPs using 2 survey tools describing HIT maturity and care environments from each discipline, respectively. To prepare the protocol, the research team conducted 3 iterative focus groups with clinicians (NPs and NH experts), recruiters from 2 national survey teams experienced with recruitment in NHs and with NPs, and a statistician to achieve consensus on which mixed-mode designs to incorporate into this research. Our research protocol workflow is illustrated in Figure 2 . The following sections include descriptions of the mixed-mode workflows by discipline and the surveys being used in this protocol.

Survey 1: NH Administrator and HIT Maturity

For each randomly selected NH, contact information for NH Administrators has been obtained using IQVIA data set. Our team searched NH websites to confirm contact information of current administrators. During initial contact with each NH administrator (either by phone or a mailed letter), we describe the study’s purpose and explain the study. All administrators who are contacted and agreed to participate in the study will be sent a cover letter providing details about the study’s purpose, instructions on how to complete the NH HIT maturity survey tool, and descriptions of the benefits and risks of participation. We provide a description of the HIT maturity survey for administrators including that the survey measures HIT capabilities, extent of HIT use, and degree of HIT integration in resident care, clinical support, and administrative activities [ 9 ]. We incorporate 2 mixed-mode designs when recruiting NH administrators including a Delayed Concurrent Mode and a Sequential Mode with regular follow-up phone contacts to stimulate engagement.

Delayed Concurrent Mode

Our primary mode for this study is a Delayed Concurrent mixed-mode design. In this mode, administrators are offered the choice between multiple modes. During the first contact (conducted by phone), we describe the project and obtain email addresses for administrators who agree to participate. Then, we follow up with administrators by email with an electronic survey link and a PDF simultaneously. This is important because the different choices among electronic surveys and PDFs allow administrators the flexibility to choose a mode that fits their needs. In nonresponse cases, administrators are later offered a different mode including a postal letter with a QR code that includes a URL link to the survey tool that is subsequently sent at a later time.

Sequential Mode

As a secondary option, we incorporate a Sequential Mode for a minimum of 10% of the facilities in each state. In this mode, participants are offered only 1 mode at a time and only part of the nonrespondents are invited for the second mode. The first mode includes mailing a postal letter that describes the study and provides both a QR code and URL link to the survey for the NH administrator. Recruiters make a series of follow-up calls to administrators after the letter is sent. During follow-up calls, emails are confirmed by the recruitment team. In this sequence, administrators who agree to take the survey and have provided their email addresses but have failed to respond with a completed mailed or faxed survey after a minimum of four follow-up calls are offered a second mode. The second mode includes a URL link and a PDF of the survey that is sent to administrators via email.

Survey 2: NPs and Care Environment

The recruitment team asks administrators to confirm that at least one NP works in their facility (whether employed by the NH or by an external health organization that provides NP services to the facility) and to verify the NP’s name and contact information. NP’s contact information listed in the IQVIA data is confirmed with administrators to ensure that it is current. Contact information that is not current is updated by the recruitment team in the recruitment database. NHs that do not meet the eligibility criterion (eg, NP left and no new NP hired) are excluded. The research team will incorporate a concurrent mixed-mode design to recruit NPs for the study.

Concurrent Mode

NPs are contacted by email or phone by our recruitment team and are provided with information describing the study, its voluntary nature, and confidentiality per the institutional review board’s (IRB’s) protocol. NPs are sent links to both an electronic survey and PDF concurrently. We expect some NHs to have more than one NP complete a survey.

Ethical Considerations

The protocol was approved by the IRB (AAAU3845). Ethical issues that were addressed in our IRB protocol included confidentiality and anonymity of privacy to encourage honest responses. Security and accessibility of the data only to authorized research staff. Researchers also created plans for minimizing coercive behaviors during recruitment (eg, applying pressure) by creating systematic follow-up and templates with recruitment language to use during contacts. The research protocol and all procedures were approved by Columbia University IRB (AAAU3845).

Follow-Up and Engagement

Up to 4 follow-up phone calls are conducted at specified 2-week intervals for administrators who have agreed to participate. Administrators and NPs who do not complete surveys are marked as “No Contact.” Administrators and NPs who complete a survey receive US $25 compensation in the form of a gift card.

Survey Coding and Cleaning

All survey data collection is conducted through REDCap (Research Electronic Data Capture; Vanderbilt University) a web-based software designed for data collection and management in research studies with emphasis on data security and flexibility [ 27 ]. We maintain data about recruitment efforts in REDCap, including number of facilities contacted, persons contacted at each facility, packets or links sent, surveys received, initial cannot reach, contact calls made, follow-up calls made, confirmations received (will complete and not completed), stated completions, and follow-up cannot reach. Recruitment staff, including a project coordinator and 4 research assistants, make recruitment calls and send surveys to NH administrators and NPs.

Data collected via electronic survey are electronically transferred to the REDCap database. Data collected via PDF is manually entered into the REDCap system by our research staff. A meticulous data-cleaning strategy is used before formal statistical analysis to ensure the data quality [ 28 ]. We used algorithms to check questionnaires for consistency and validity. For example, graphical exploration through boxplots, histograms, and scatter plots will be used to help with detecting outliers and logically implausible data points. Any identified outlying observations undergo thorough examination to discern between potential data entry errors and genuinely extreme values. Data entry errors are corrected. Any systematic patterns will be scrutinized. Every step of the data cleaning process and associated decisions are documented to ensure transparency.

There is a possibility that some NH administrators or NPs who agree to participate in the study will not fill out an HIT maturity or care environment survey tool completely. We anticipate that there may be some missing data on completed surveys. Based on prior national HIT maturity and NP studies, we have estimated that less than 3% of the data for surveys received was missing for both types of surveys. We plan to use all available data in our analyses.

Survey Measures

NH HIT Maturity [ 8 , 29 ] is measured using a total composite score that correspond to 7 HIT maturity stages. The 7 maturity stages range from the lowest HIT Maturity Stage 0—Nonexistent HIT solutions or electronic health records to Stage 6—Use of data by residents and resident representatives to generate clinical data and drive self-management. A higher total HIT maturity score indicates greater IT capabilities, use, and integration in resident care, clinical support (including IT systems in pharmacy, radiology, laboratory), and administrative activities in the NH. The overall standardized Cronbach α for this instrument in past research was 0.86 (high); each dimension or domain achieved a Cronbach α ranging from 0.7 to 0.9 [ 30 ].

NP Care Environment is measured by a 44-item Nurse Practitioner Nursing Home Organizational Climate Survey (NP-NHOCS) [ 31 ], which asks NPs to rate the work attributes in NHs using a 5-point Likert scale. The NP-NHOCS has five subscales: (1) NP-Physician Relations (7 items)—measures the relationship, communication, and teamwork between NPs and physicians; (2) NP-Administration Relations (11 items)—measures collaboration and communication between NPs and managers; (3) NP-Director of Nursing Relations (8 items)—measures the relationship, communication, and teamwork between NPs and Directors of Nursing; (4) Independent Practice and Support (9 items)—measures resources and support NPs have for their independent practice; and (5) Professional Visibility (9 items)—measures how visible the NP role is in the organization. We first compute NP-level and then NH-level mean scores by aggregating the responses of all NPs in the NHs as recommended [ 32 ]. Higher mean scores indicate better care environments. NPs are asked to complete measures of demographics (eg, age, sex, and experience).

A number of planned analyses will be performed. In terms of HIT maturity survey, we aim to understand which survey mode (Delayed Concurrent vs Sequential) will maximize NHs’ engagement in our research project and which factor(s) influence survey completion method. First, descriptive statistics will be used to summarize the key variables of interest including but not limited to response rates (agreeing to participate or not), completion rates, time taken to complete the survey, and the proportion of electronic surveys received. Chi-square test or Fisher exact test will be used to examine differences in response rates, completion rates, proportion of electronic survey received between the NHs assigned to the Delayed Concurrent Mode and Sequential Mode survey designs. This analysis will determine if one survey mode yields higher response and completion rates compared to the other. Second, if there is sufficient data availability, linear regression models will be used to test whether NH administrators’ demographic characteristics (ie, age, sex, race or ethnicity), NH-level characteristics (eg, bed size and staffing hours), and HIT maturity level are associated with the choice of survey completion method (electronic or PDF format).

In terms of NP care environment survey, all NPs will be offered both an electronic survey and a PDF concurrently. The proportion of electronic surveys received among those who respond will be calculated to determine preference for electronic over PDF surveys. If a sufficient number of electronic and PDF surveys are received, linear mixed effects models with NH as random effect will be used to assess whether the choice of survey completion method is associated with NH-level characteristics (eg, HIT maturity score, geographical location, ownership), NP-level characteristics (eg, age, race or ethnicity, years of experience, and job roles), and NP care environment scores, respectively.

The research team conducted 3 iterative focus groups with a total of 14 clinicians including NPs and survey recruitment experts. The following pros and cons were used to determine our recruitment strategies.

Pros of Mixed-Mode Designs

The pros of mixed-mode designs identified by the team during focus groups were that delayed concurrent, concurrent, and sequential mixed-mode approaches can save recruitment time compared to single mode delivery methods. Additionally, effort on the part of recruitment staff is minimized by using mixed-modes. By using mixed survey modes, participants can immediately choose their preferred survey method, potentially enhancing their satisfaction with the survey process. This facilitates engagement that leads to completed surveys and increased response rates. Another pro of the concurrent mode identified was that sending a QR code via the postal service in addition to providing a URL link provides greater selectivity and plasticity in a respondent’s choice, which could enhance engagement and responsiveness to surveys. A pro of single mode designs is the potential for quick turnaround times and representative samples for projects with limited resources [ 33 ].

Cons of Mixed-Mode Designs

One disadvantage of single mode strategies is that they decrease the versatility and adaptability to different organizational capabilities (eg, access to email and system firewalls), which could reduce response rates. For example, a URL link sent via email might be more difficult for NH administrators and NPs to open due to system firewalls put into place by organizations to meet higher level security standards of HIT systems. We identified another con of a sequential mode; for instance, if a singular mode is offered when initial recruitment is started, the respondent may not engage in the second wave. For example, if respondents are concerned about access to email, they may not engage with us again in further calls if the first mode offered including email is perceived as a barrier to participation. Other cons that were identified related to NH infrastructure and environmental variables. For instance, NPs might have limited access or no workspace available to print a PDF and to complete a survey. Other reported cons of mixed-mode designs (sequential modes [web then telephone]) compared to single mode (telephone only) include higher missing data rates and more focal responses [ 34 ].

After randomization, we rigorously compared selected and nonselected NHs based on key NH level characteristics such as bed size, ownership, location, staffing hours, payer mix, and overall rating. Our analysis did not reveal statistically significant differences in these characteristics (See Table S1 in Multimedia Appendix 1 ).

The research study was funded in February 2023. Participant recruitment for the project began in June 2023. As of June 3, 2024, a total of 109 HIT maturity surveys and 83 NP surveys have been returned. About 69% of the HIT maturity surveys have been submitted using the electronic link and 27% were submitted after a QR code was sent to the administrator. About 95% of the NP surveys were returned with electronic survey links.

Our national study is the first to our knowledge to focus on NH HIT maturity and NP care environments where administrators and NPs work. Although NPs are a predominant provider in NHs [ 35 ], no study to date has focused on NP care environments and available resources (eg, technology) to this discipline, leading to limited understanding of how NPs conduct work, and how HIT maturity contributes to an NP’s ability to improve care and outcomes for NH residents with serious chronic conditions. Furthermore, a primary objective of this study is to provide evidence of how administrators and NPs codesign technologies that can transform care delivery in NHs. Our team anticipates that using mixed-modes will enhance our ability to work with participants at different stages of HIT maturity, which we believe is in an important factor in how care environments are perceived by employees (eg, NPs) in these settings.

To achieve this goal, we first must be able to maximize engagement in this survey research with strong representation by both NH administrators and NPs from all US states. Second, we must mitigate barriers to NH administrators and NPs accessing surveys so that they can participate. Finally, we must achieve acceptable response rates by generating different modes of support, providing choice and flexible means for NH administrators and NPs to participate in the survey process. In this protocol, we have identified mixed-mode recruitment strategies based on the expert opinion of experienced survey recruitment staff that should enable us to meet our goals and to achieve a representative national sample of NH administrators and NPs.

Limitations

This study may have limitations. In prior work, we have identified great variability in HIT capabilities among many NH's, such as access to external email and connectivity challenges where NH staff work [ 36 ]. Depending on the survey mode used during the data collection, this variation may create differences in response rates between facilities. We have incorporated various mixed-mode methods in this research protocol that should allow respondents to choose their preferred method and the ability to complete a survey considering their institutional characteristics. The use of mixed-modes has been shown to improve participation in survey research, thus reducing barriers for less well-resourced NHs (eg, NHs with lower HIT maturity levels). Less resourced NHs are typically those with greater resident ethnic and racial diversity [ 37 ], so improving their participation is critical to enhance representation of these communities, which is a benefit of the design.

Conclusions

This research protocol describes a study using 2 survey tools to measure HIT maturity and NP care environments in the US NHs as perceived by administrators and NPs. We have identified the pros and cons of survey recruitment strategies experienced by our team in past work. We reviewed evidence-based recruitment strategies using mixed-modes which are defined in the literature as methods that incorporate the use of 2 or more modes to recruit respondents. In this protocol, we have incorporated a delayed concurrent mode, sequential mode, and a concurrent mode to enhance engagement, mitigate barriers to survey access, and to increase response rates in collecting survey data both from NH administrators and NPs to have robust data for future analysis.

Acknowledgments

The authors wish to acknowledge Dr Richard Chan and Ms Hana Amer for the contributions in the initial stages of determining the steps in the research protocol. Research reported in this publication was supported by the National Institute on Aging of the National Institutes of Health (award R01AG080517). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Authors' Contributions

GLA, LP, YZ, MH, SK, AAN, KW, MBS, AK, TB, and ST contributed to the design, acquisition, interpretation, writing, and revision of this manuscript.

Conflicts of Interest

None declared.

Supplementary Table.

• Silverman HD, Steen EB, Carpenito JN, Ondrula CJ, Williamson JJ, Fridsma DB. Domains, tasks, and knowledge for clinical informatics subspecialty practice: results of a practice analysis. J Am Med Inform Assoc. 2019;26(7):586-593. [ FREE Full text ] [ CrossRef ] [ Medline ]
• De Groot K, De Veer AJE, Paans W, Francke AL. Use of electronic health records and standardized terminologies: a nationwide survey of nursing staff experiences. Int J Nurs Stud. 2020;104:103523. [ CrossRef ] [ Medline ]
• Jones TL, Baxter MAJ, Khanduja VA. A quick guide to survey research. Ann R Coll Surg Engl. 2013;95(1):5-7. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abramson EL, McGinnis S, Moore J, Kaushal R, HITEC investigators. A statewide assessment of electronic health record adoption and health information exchange among nursing homes. Health Serv Res. 2014;49(1 Pt 2):361-372. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jones CD, Jones J, Bowles KH, Flynn L, Masoudi FA, Coleman EA, et al. Quality of hospital communication and patient preparation for home health care: results from a statewide survey of home health care nurses and staff. J Am Med Dir Assoc. 2019;20(4):487-491. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Giai J, Boussat B, Occelli P, Gandon G, Seigneurin A, Michel P, et al. Hospital survey on patient safety culture (HSOPS): variability of scoring strategies. Int J Qual Health Care. 2017;29(5):685-692. [ CrossRef ] [ Medline ]
• Van Vaerenbergh Y, Thomas TD. Response styles in survey research: a literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research. 2012;25(2):195-217. [ CrossRef ]
• Alexander GL, Powell K, Deroche CB, Popejoy LL, Mosa ASM, Koopman R, et al. Building consensus toward a national nursing home information technology maturity model. J Am Med Inform Assoc. 2019;26(6):495-505. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alexander GL, Deroche CB, Powell KR, Mosa ASM, Popejoy L, Koopman RJ, et al. Development and pilot analysis of the nursing home health information technology maturity survey and staging model. Res Gerontol Nurs. 2022;15(2):93-99. [ CrossRef ] [ Medline ]
• Krosnick JA, Presser S, Husbands-Fealing K, Ruggles S. The future of survey research: challenges and opportunities. Arlington VA: The National Science Foundation. 2015:1-163.
• Schouten B, Brakel JVD, Buelens B, Giesen D, Luiten A, Meertens V. Designing Mixed-Mode Surveys. Mixed-Mode Official Surveys: Design and Analysis. Boca Raton FL. CRC Press; 2022.
• Sammut R, Griscti O, Norman IJ. Strategies to improve response rates to web surveys: a literature review. Int J Nurs Stud. 2021;123:104058. [ CrossRef ] [ Medline ]
• Sastry N, McGonagle KA. Switching from telephone to web-first mixed-mode data collection: results from the transition into adulthood supplement to the US panel study of income dynamics. J R Stat Soc Ser A Stat Soc. 2022;185(3):933-954. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Patrick ME, Couper MP, Jang BJ, Laetz V, Schulenberg JE, O'Malley PM, et al. Building on a sequential mixed-mode research design in the monitoring the future study. J Surv Stat Methodol. 2022;10(1):149-160. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schouten BBJVD, Buelens B, Giesen D, Luiten A, Meertens V. Foreward to Mixed-Mode Official Surveys: Design and Analysis. Mixed-Mode Official Surveys: Design and Analysis. Boca Raton FL. CRC Press; 2022:3-8.
• VanGeest J, Johnson TP. Surveying nurses: identifying strategies to improve participation. Eval Health Prof. 2011;34(4):487-511. [ CrossRef ] [ Medline ]
• Ellis LA, Pomare C, Churruca K, Carrigan A, Meulenbroeks I, Saba M, et al. Predictors of response rates of safety culture questionnaires in healthcare: a systematic review and analysis. BMJ Open. 2022;12(9):e065320. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beebe TJ, Jacobson RM, Jenkins SM, Lackore KA, Rutten LJF. Testing the impact of mixed-mode designs (Mail and Web) and multiple contact attempts within mode (Mail or Web) on clinician survey response. Health Serv Res. 2018;53 Suppl 1(Suppl Suppl 1):3070-3083. [ FREE Full text ] [ CrossRef ] [ Medline ]
• CDC. National Center for Health Statistics Nursing Home Care. Atlanta Georgia. Centers for Disease Control and Prevention/National Center for Health Statistics; 2017.
• White E, Woodford E, Britton J, Newberry LW, Pabico C. Nursing practice environment and care quality in nursing homes. Nurs Manage. 2020;51(6):9-12. [ CrossRef ] [ Medline ]
• CMS. Centers for medicare and medicaid nursing home compare. Centers for Medicare and Medicaid Sept. 2019. [ CrossRef ]
• Adler-Milstein J, Raphael K, O'Malley TA, Cross DA. Information sharing practices between US hospitals and skilled nursing facilities to support care transitions. JAMA Netw Open. 2021;4(1):e2033980. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Burke RE, Phelan J, Cross D, Werner RM, Adler-Milstein J. Integration activities between hospitals and skilled nursing facilities: a national survey. J Am Med Dir Assoc. 2021;22(12):2565-2570.e4. [ CrossRef ] [ Medline ]
• Farid M, Machta RM, Jones DJ, Furukawa MF, Miller D. Nursing Homes Affiliated with U.S. Health Systems. Rockville MD. Agency for Healthcare Research and Quality; 2018. URL: https://www.ahrq.gov/sites/default/files/wysiwyg/chsp/data/chsp-brief8-nursinghomes.pdf [accessed 2024-02-07]
• CMS. Centers for Medicare and Medicaid. Washington DC. Centers for Medicare and Medicaid Special Focus Facility Program URL: https://www.cms.gov/files/document/sffpostingwithcandidatelist-october2023pdf.pdf [accessed 2023-12-09]
• Alexander GL, Kueakomoldej S, Congdon C, Poghosyan L. A qualitative study exploring nursing home care environments where nurse practitioners work. Geriatr Nurs. 2023;50:44-51. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Van den Broeck J, Cunningham SA, Eeckels R, Herbst K. Data cleaning: detecting, diagnosing, and editing data abnormalities. PLoS Med. 2005;2(10):e267. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alexander G, Deroche C, Powell K, Mosa A, Popejoy L, Koopman R. Forecasting content and stage in a nursing home information technology maturity instrument using a delphi method. J Med Syst. 2020;44(3):60. [ CrossRef ] [ Medline ]
• Poghosyan L, Nannini A, Finkelstein S, Mason E, Shaffer J. Development and psychometric testing of the nurse practitioner primary care organizational climate questionnaire. Nurs Res. 2013;62(5):325-334. [ CrossRef ] [ Medline ]
• Bono C, Ried LD, Kimberlin C, Vogel B. Missing data on the center for epidemiologic studies depression scale: a comparison of 4 imputation techniques. Res Social Adm Pharm. 2007;3(1):1-27. [ CrossRef ] [ Medline ]
• Williams JA, Vriniotis MG, Gundersen DA, Boden LI, Collins JE, Katz JN, et al. How to ask: Surveying nursing directors of nursing homes. Health Sci Rep. 2021;4(2):e304. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ofstedal M, Kézdi G, Couper M. Data quality and response distributions in a mixed-mode survey. Longit Life Course Stud. 2022;13(4):621-646. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rantz M, Popejoy L, Vogelsmeier A, Galambos C, Alexander G, Flesner M, et al. Reducing avoidable hospitalizations and improving quality in nursing homes with APRNs and interdisciplinary support: lessons learned. J Nurs Care Qual. 2018;33(1):5-9. [ CrossRef ] [ Medline ]
• Alexander GL, Steege LM, Pasupathy KS, Wise K. Case studies of IT sophistication in nursing homes: a mixed method approach to examine communication strategies about pressure ulcer prevention practices. International Journal of Industrial Ergonomics. 2015;49:156-166. [ CrossRef ]
• Sloane P, Yearby R, Konetzka R, Li Y, Espinoza R, Zimmerman S. Addressing systemic racism in nursing homes: a time for action. J Am Med Dir Assoc. 2021;22(4):886-892. [ CrossRef ] [ Medline ]

Abbreviations

Edited by S Ma; submitted 08.01.24; peer-reviewed by T Mujirishvili; comments to author 23.03.24; revised version received 22.04.24; accepted 28.06.24; published 29.08.24.

©Gregory L Alexander, Lusine Poghosyan, Yihong Zhao, Mollie Hobensack, Sergey Kisselev, Allison A Norful, John McHugh, Keely Wise, M Brooke Schrimpf, Ann Kolanowski, Tamanna Bhatia, Sabrina Tasnova. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 29.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.

• Search term Advanced Search Citation Search
• Individual login
• Institutional login

Understanding Work Rhythms in Software Development and Their Effects on Technical Performance

Jiayun Zhang

• orcid.org/0000-0002-3562-5794

Shanghai Key Lab of Intelligent Information Processing , School of Computer Science , Fudan University , Shanghai , China , fudan.edu.cn

Corresponding Author

Qingyuan Gong

• [email protected]
• orcid.org/0000-0001-7942-8752

Research Institute of Intelligent Complex Systems , Fudan University , Shanghai , China , fudan.edu.cn

• orcid.org/0000-0003-4749-3060
• orcid.org/0000-0002-4517-3779

Department of Information and Communications Engineering , Aalto University , Espoo , Finland , aalto.fi

• orcid.org/0000-0002-9405-4485

Aaron Yi Ding

• orcid.org/0000-0003-4173-031X

Department of Engineering Systems and Services , Delft University of Technology , Delft , Netherlands , tudelft.nl

The temporal patterns of code submissions, denoted as work rhythms, provide valuable insight into the work habits and productivity in software development. In this paper, we investigate the work rhythms in software development and their effects on technical performance by analyzing the profiles of developers and projects from 110 international organizations and their commit activities on GitHub. Using clustering, we identify four work rhythms among individual developers and three work rhythms among software projects. Strong correlations are found between work rhythms and work regions, seniority, and collaboration roles. We then define practical measures for technical performance and examine the effects of different work rhythms on them. Our findings suggest that moderate overtime is related to good technical performance, whereas fixed office hours are associated with receiving less attention. Furthermore, we survey 92 developers to understand their experience with working overtime and the reasons behind it. The survey reveals that developers often work longer than required. A positive attitude towards extended working hours is associated with situations that require addressing unexpected issues or when clear incentives are provided. In addition to the insights from our quantitative and qualitative studies, this work sheds light on tangible measures for both software companies and individual developers to improve the recruitment process, project planning, and productivity assessment.

1. Introduction

The time allocation for work activities is closely related to a software developer’s daily routine and reflects her/his work habits. We define the work rhythms in the process of software development as the temporal patterns shown in developers’ code submission activities. A typical work rhythm of a developer could be described as follows: the developer may start the work at 9 a.m. on working days and concentrate on writing and submitting code during working hours. She/he would take a short break at noon for lunch and the code submissions could stop for a while as well. After finishing the tasks at 6 p.m., the codes will not be updated until 9 a.m. on the next working day. Developers working in companies with diverse cultures follow different work rhythms. It was reported that one third of software developers do not adopt a typical working hour rhythm (e.g., from 10 a.m. to 6 p.m.) [ 1 ]. The issues of developers’ work rhythms have been discussed extensively. Some Chinese tech companies have adopted an unofficial work schedule known as the “996 working hour system,” which requires employees to work from 9 a.m. to 9 p.m., 6 days a week. The public quickly took notice of these extreme working hours as they were shared on social media ( https://github.com/996icu/996.ICU ). This abnormal work schedule has received criticism, arguing that developers cannot keep focusing on programming during such long working hours and their efficiency and productivity decrease after working for long hours ( https://www.scmp.com/tech/start-ups/article/3005947/quantity-or-quality-chinas-996-work-culture-comes-under-scrutiny ). However, global leading news media, such as Cable News Network (CNN; https://edition.cnn.com/2019/04/15/business/jack-ma-996-china/index.html ) and British Broadcasting Corporation (BBC) News ( https://www.bbc.com/news/business-47934513 ), reported another voice that many successful entrepreneurs weighed on the advantages of long-hour work schedules to the companies. These heated discussions with controversial perspectives press an urge demand to understand developers’ work rhythms and their effects on practical technical performance.

Studying work rhythms in software development yields many important implications. For example, the profiles and activities in online developer communities are considered as reliable indicators of technical performance during the hiring process [ 2 ]. However, having more commits during off-hours does not necessarily equate to better code quality. Instead of assessing based on the quantity of commits, it is crucial to acquire a deeper understanding of work rhythms and their effects. Such insights can help employers gain deeper knowledge about job applicants’ work habits before hiring. In addition, software development teams can rely on more rational assessments of technical performance rather than judging merely by the time spent in the office. With an understanding of the effects of work rhythms on technical performance, both project teams and individual developers can better allocate and schedule their time in development.

The existing studies on the work rhythms of people in different occupations often cover their effects on work performance. Alternative work schedules, such as flexible and compressed work schedules, had positive effects on work-related criteria including productivity and job satisfaction [ 3 , 4 ]. Conversely, sustained work during long working hours was associated with an increased risk of errors and decreased work performance [ 5 , 6 , 7 , 8 , 9 ]. In the field of software engineering, multiple studies have examined the relationship between code quality and the time when the work is performed. It has been found that the bugginess of commits is related to the time (i.e., the hour of the day) when those commits have been made, but there are large variations among individuals and projects [ 10 , 11 , 12 ].

Previous studies have primarily focused on the effects of work hours on code quality, within the contexts of limited organizations and have primarily considered code bugginess as a quality metric. In addition, they have not sufficiently addressed the circadian and weekly patterns that characterize developers’ work habits. Our study leverages a large-scale real-world dataset from GitHub to explore how work rhythms correlate with multiple dimensions of technical performance. Considering that project-level working behaviors often involve collaborative efforts of multiple contributors and do not necessarily reflect the work patterns of individual developers, our study analyzes both project- (in our study, the term “project” is used synonymously with “repository”) and individual-level metrics. We aim to provide a more comprehensive understanding of work patterns from two different yet interconnected perspectives. Specifically, we apply spectral biclustering [ 13 ] to identify the work rhythms from both the individual and project perspectives. The biclustering algorithm simultaneously groups both rows and columns of a data matrix, allowing us to understand the groups of similar subjects (i.e., developers/repositories) and their typical commit behaviors at the same time. We analyze the relationship between the identified work rhythms and demographics (such as region and account/repository age) and collaboration roles (i.e., whether a developer is a structural hole spanner (SHS) [ 14 ]). We use popularity metrics (such as followers, stars, forks, and issues on GitHub) and code productivity (measured by lines of code changed per week) as indicators of technical performance. Then, we perform a comprehensive analysis to investigate how these work rhythms influence technical performance. Furthermore, we conduct a survey study to complement the results of empirical data analysis.

We design an approach with spectral biclustering algorithm to identify the work rhythms of repositories and individual developers. This method reveals four distinct work rhythms among individuals and three among repositories.

We present an empirical analysis of the correlations between work rhythms and demographics including regions, age, and collaboration roles. We define multiple practical measures for technical performance and study the effects of work rhythms on them.

We conduct a survey involving 92 respondents to gain insights into developers’ experiences and the reasons and attitudes towards overtime work.

We introduce the background and related works in Section 2 and research questions in Section 3 , followed by our research methods (Section 4 ) and results (Section 5 ). We discuss the significance of our contributions in Section 6 and offer some concluding remarks in Section 7 .

2. Background and Related Work

Developers are engaged in multiple work activities in a given week and follow some rules in the time usage in software development [ 15 , 16 , 17 ]. Sequential analysis of the generated contents is crucial for understanding the behavior patterns of online users [ 18 , 19 ]. The widely used development tools such as version control systems and online developer communities ensure the transparency of the workflows, which provide researchers with abundant resources to investigate developers’ work practices [ 20 , 21 , 22 , 23 ]. By exploring the data from these development tools, multiple studies have examined developers’ work practices and contributions.

First, the work time in software development has been studied. For example, Claes et al. [ 1 ] defined work rhythm as the circadian and weekly patterns of commits. They analyzed the commit timestamps of 86 open source software projects and reported that two-thirds of the developers follow a standard work schedule and rarely work nights and weekends. In addition, Traulle and Dalle [ 24 ] investigated the evolution of developers’ work rhythms. They observe a trend where developers adopt more regular work patterns over time and start working increasingly earlier. Furthermore, this study is related to our previous work [ 25 ], which examined the commit activities of tech companies in China and the United States and compared the differences in working hours between companies in the two countries. Compared with our previous work, this study expands the scope and introduces new research questions—the correlations between work rhythm and technical performance. In addition, we enlarge the dataset to include a wider range of regions and approach the analysis of working behaviors at more granular levels by examining both project- and individual-level behaviors.

Second, the relationships between work quality and work time have been investigated. For example, Khomh et al. [ 26 ] studied the impact of Firefox’s rapid release cycle on software quality. They found that the fast release cycle did not lead to more bugs but accelerated the process of fixing bugs. In addition, several studies focused on the relationships between the bugginess of code and the hour of the day when the code is submitted. For instance, Eyolfson et al. [ 10 , 11 ] studied three well-known open source projects and found that more bugs are contained in commits made during midnight and early morning, while commits made in the morning have the best quality. Prechelt and Pepper [ 12 ] investigated a closed-source industry project and proposed that 8 p.m. is the hour with the highest error rate. It is observed that results vary across different projects.

Previous research on the effects of work time often investigates projects from limited organizations and only considers the bugginess of code as the metric of code quality. In addition, these studies typically focus on the effects of specific hours of the day, rather than the circadian and weekly patterns. There is no sufficient investigation with solid evidence yet to show the relationship between work rhythms and technical performance from multiple aspects. In this paper, we perform data analysis on a real-world code submission dataset collected from GitHub, a prominent online developer platform with more than 100 million developers and hosting more than 420 million repositories ( https://github.com/about , accessed on May 18, 2024).

During the software development, people often use Git, a distributed version control system, to monitor the modifications to the code. To submit code changes to Git, people make commits that include details such as authorship, timestamp, and the code changes made. The temporal distribution of a developer’s commit logs reflects her/his rhythm of submitting code changes. These commit logs can be accessed if the projects are uploaded to GitHub and set to publicly visible. Figure 1 shows the time distribution of developers’ code submissions on GitHub. The statistics are generated according to the GitHub User Dataset [ 27 , 28 ]. The dataset consists of the information and activities about more than 10 million randomly selected GitHub users. We focus on the users who have more than 100 commits and have submitted codes on more than 100 different days. Among these users, we select 13,201 of the developers with 5,406,933 commits. In general, developers commit more frequently on weekdays than at weekends. There are peak hours of code submissions at 11 a.m., 4 p.m., and 10 p.m., and an off-peak period during the early morning, which conforms to the common sense of people’s daily life. The aggregated commit logs in Figure 1 show that developers exhibit temporal regularities in code submissions. However, given the differences in the adoption of work practices, such general work rhythm could not represent effectively the work habit of each developer.

3. Research Questions

We aim to study the work rhythms of developers and software projects to have a comprehensive view of work rhythms in software development from both the individual and group levels. Our study is guided by the following four research questions:

RQ1. What are the work rhythms of individual developers and software projects?

RQ2. Are work rhythms related to demographics and collaboration roles?

The first two RQs intend to reveal representative work rhythms among individual developers and software projects and examine discrepancies in the demographics of the developers with different work rhythms.

RQ3. What are the correlations between different work rhythms and technical performance?

The third RQ is to seek a deeper understanding of the relationships of different work rhythms with the outcome of work by considering various metrics for technical performance.

RQ4. What are developers’ attitudes towards work rhythms and productivity?

The last RQ investigates developers’ actual work experience and their views on productivity.

In this section, we present the data collection and analysis methods in our study. A summary of the research subjects, variables, and the methods of data analysis for each research question is provided in Table 1 . The overview of the methodology is presented in Figure 2 .

4.1. Data Collection

The commit logs of public projects on GitHub are publicly visible and can be retrieved using the GitHub API. Our data collection adhered to “terms of service” of GitHub ( https://help.github.com/articles/github-terms-of-service/ ). The data collection took place from May 1 to May 27, 2019. The dataset covers the commit activities of the source repositories of 110 organizations ever since the repositories were created. The location of the companies spread a wide range from the United States (such as Facebook, Amazon, and Google) to China (such as Baidu, Tencent, and Alibaba) and Europe (such as SAP, Nokia, and Spotify). To accurately assess work rhythms, we used the local time of each commit log to avoid the potential influence of different time zones in which the commits were made. Commit logs without time zone information (9.03% of the total) were excluded. Following the data cleaning, a total of 1,532,439 commits remained. Then, we group these commits by repositories and committers respectively and form the following two datasets for our analysis.

Company repositories. We scanned the repository lists of the 110 organization accounts and crawled descriptive information about the repositories and commit logs submitted into the repositories. We selected repositories with at least 300 commits and formed the repository dataset with a total of 1,131 repositories and 1,111,685 commits.

Individual developers. To study the work rhythms of individual developers, we first merged different identities of the same developer, as a developer may have multiple identities on GitHub and in the version control system. We extracted the email from the version control system’s author field and GitHub account ID from the author field recorded in GitHub commit activity. We created a mapping from email addresses to GitHub accounts and grouped together identities that shared the same account ID or email address. Following this dealiasing process, 47.1% of the committer identities were merged. Then, we chose the core developers by selecting those with at least 30 commits. These developers are the top 12.5% of the committers and have made 85% out of all commits in our dataset. We further crawl the GitHub account information of the developers, including number of followers and number of stars in each of their own repositories. Finally, we formed our developer dataset with 7,509 individual developers and 1,296,715 commits, among which 2,754 have detailed information about their GitHub accounts.

4.2. Identifying Work Rhythms

To profile how commits are created by a developer or in a project repository, we compute the frequencies of commit activities across different time intervals and apply clustering to identify patterns.

4.2.1. Data Processing

4.2.2. biclustering model.

Among various classical clustering methods, such as K-means [ 30 ], DBSCAN [ 31 ], and the state-of-the-art ones designed for specific applications such as topic models (latent Dirichlet allocation) [ 32 , 33 ], we choose the spectral biclustering [ 13 ] algorithm to discover the work rhythms in our dataset. Spectral biclustering is a clustering technique, which generates biclusters—a group of samples (in row) that show similar behavior across a subset of features (in column), or vice versa. In our scenario, we group both developers/repositories and the commit behavior at a time to understand the groups of similar subjects and their typical behaviors. Specifically, developers/repositories grouped in different row clusters show different commit behaviors. In addition, the column clusters outputted by the algorithm enable us to infer how developers/repositories in different row clusters behave in each subset of hours. Developers/repositories with the same rhythm have similar commit frequencies in each subset of hours.

The model takes the 48-dimensional vectors as input and automatically discovers the clusters of work rhythms by measuring the similarities between them. To implement the clustering model, we used Scikit-learn [ 34 ], a widely used machine-learning library. To determine the optimal parameter setting, we perform an iterative search for the number of work rhythms k from 2 to 8 with empirical experiments. For each k , we visualize the rhythms and examine the number of samples in clusters to ensure that the clusters have sufficient individuals and exhibit distinct patterns beyond mere time shifting. We choose k as the largest value among those tested that yields stable and distinctive work rhythms.

4.3. Empirical Analysis on Identified Work Rhythms

4.3.1. demographics of developers and repositories.

We intend to explore whether developers or repositories with specific demographic information tend to follow specific work rhythms.

First, local cultures may have an impact on work rhythms. To investigate whether there is a difference among developers who work on repositories from different regions in terms of work rhythms, we examine the countries of the repositories that the developers worked on. For each developer, we group the repositories that she/he has made contributions to and check which countries the organizations of the repositories belong to. If a developer has contributions to repositories from more than one country, we set the work region of the developer as “multiple countries.” We target four different regions: the United States, China, Europe, and multiple countries.

In addition, considering the fact that senior developers may take charge of more projects than junior developers, we assume that senior developers have different work rhythms from young developers. For this purpose, we investigate whether there is a correlation between the type of work rhythms and the seniority of the developers. We use the number of days after the creation time of GitHub account as a proxy for one’s seniority in programming.

Furthermore, according to Vasilescu et al.’s [ 35 ] study, there are differences in terms of productivity between younger repositories and older ones. As a result, repositories with longer histories may have different work rhythms from newly created ones. We count the number of days since a project was created on GitHub as the measure of repository age.

4.3.2. Collaboration Role

Collaboration is an important feature of software engineering. The developer’s participation in project collaboration is a testament to her/his technical ability.

The structural hole theory [ 14 , 36 , 37 , 38 ] in social network analytics suggests that people who are positioned in structural holes, known as SHS, play a critical role in the collaboration and management of the teams. A structural hole is perceived as a gap between two closely connected groups. SHS fill in the gaps among different groups. They control the diffusion of valuable information across groups and come up with new ideas by combining ideas from multiple sources [ 14 ]. Bhowmik et al. [ 39 ] studied the role of structural holes in requirements identification of open-source software development and found that structural holes are positively related to the contribution of a larger amount of new requirements and play an important role in new requirement identification.

We intend to see whether there is a difference in terms of work rhythms between SHS developers and ordinary developers. We build a collaboration graph using our dataset, in which the node represents a developer and an edge between two nodes represents the two developers have committed to the same repository. We apply an advanced SHS identification algorithm called APGreedy [ 40 ] (there are several SHS identification algorithms [ 37 , 41 , 42 ] and APGreedy is a representative one) to find the SHS in the collaboration graph and choose the top 500 developers as the SHS developers. After filtering out developers with less than 30 commits, we obtain 246 SHS developers in total. Accordingly, we select 246 non-SHS developers from the rest using random sampling to represent the ordinary developers.

4.3.3. Developer-Level Measures on Technical Performance

We define the following measures for evaluating the technical performance of a developer:

Average number of stars. GitHub provides starring function for users to mark their interest in projects. We count the average number of stars received by the repositories owned by the developer. Receiving more stars indicates a higher popularity of a project [ 43 ].

Number of followers. We use the number of followers a GitHub user has at the time of data collection as a signal of standing [ 44 ] within the community. Users with lots of followers are influential in the developer community as many people are paying attention to their activities.

H-index of Stars. The h-index [ 45 ] was originally introduced as a metric to evaluate both the productivity and citation impact of a scholar’s research publications. It has been used to measure the influence of users’ generated contents in social networks [ 46 ]. We define h-index of a developer as the maximum value of c such that the given developer has published c repositories that have each been starred at least c times. We use this metric to measure both the productivity and influence of a developer on GitHub.

4.3.4. Repository-Level Measures of Technical Performance

To examine the technical performance of repositories, we define the following measures:

Number of stars. We use the number of stars a repository has received to evaluate the popularity of a repository. A repository with many stars implies that many people show their interests in it [ 35 , 47 ].

Number of forks. The “forking” function on GitHub enables developers to create a copy of a repository as their personal repository and then they can make changes to the code freely. Similar to the number of stars discussed above, the number of forks a repository has received is another important indicator that a repository is popular [ 35 , 44 , 48 ].

Number of open issues. Issues can be used to track bugs, enhancements, or other requests. In cases where the project’s problem was suspect, submitters and core members often engaged in extended discussions about the appropriateness of the code [ 49 , 50 ]. Repositories with more open issues receive more attention than those with less.

Lines of code changed per week (LOC changed ). This measure is defined as the average number of lines of code changed (the sum of additions and deletions) in all commits in a repository per week. It is a measure of outputs produced per unit time, which serves as a proxy for productivity [ 35 , 51 , 52 , 53 ].

4.3.5. Hypothesis Testing

To accurately identify behavioral differences among different populations, we conduct statistical hypothesis testing on different groups.

First, we conduct Pearson’s chi-squared test [ 54 ] to examine if there are significant differences in the work rhythms among different groups (i.e., regions and collaboration roles) of projects or developers. The Pearson’s chi-squared test is commonly used for evaluating the significance of the association between two categories in sets of categorical data.

Second, we statistically validate if there are significant differences in the demographics and technical performance among different groups of software projects and developers. We compute the measures of each subject within the group and the measures of the population outside the group. Then, we apply the Mann–Whitney U test [ 55 ], which is commonly used to determine whether two independent samples are from populations with the same distribution.

The results of Pearson’s chi-squared test and Mann–Whitney U test are measured by p -value, where a smaller p -value indicates higher significance level in rejecting the null hypothesis H 0 . A  p -value below 0.05 indicates a significant difference among the two populations in terms of the selected measure. Cramer’s V and Cliff’s delta effect size are used to supplement the results of Pearson’s chi-squared test and Mann–Whitney U test, respectively.

4.4. User Survey

To investigate how developers experience and think of their work rhythms and productivity, we designed an online survey and sent it to developers in selected tech companies. The selected companies included a mix of large corporations and startups.

Our survey was reviewed and approved by the Institute of Science and Technology, Fudan University. Prior to the launch of the survey, we invited seven developers from different tech companies and did a pilot test. These participants completed the questionnaire and provided feedback, which we used to refine the survey. Next, we performed an undeclared pilot test involving 10 participants from selected companies in our dataset. We reviewed and discussed their responses to ensure that the questionnaire was free of major issues. After finalizing the survey, we distributed it online and asked the pilot participants to share the link to the survey with others. The survey had 1,516 views and received 92 responses from eligible respondents who identified their current job as software development. The survey questions are given in the appendix.

First, to validate our result on work rhythms, we asked survey participants about their required working hours and actual working hours on a typical work day. The participants are required to provide both their required and actual start time and end time of work or to implicate there is no required working hours.

Next, we asked participants about the time they spent on different work activities and programming themes. According to Meyer et al. [ 56 ], developers primarily identified coding-related tasks as productive, whereas activities such as attending meetings, reading, and writing emails were often considered as unproductive. To gain insight into productivity both during and outside office hours, we asked participants to indicate the percentage of time they spent on various work activities during these periods, including coding, studying, project planning, writing documents, contacting colleagues, meeting, social activities, and others. Participants could choose one among the following five options to indicate the percentage of time they spent on each work activity or programming theme: “less than 5%,” “between 5% and 20%,” “between 20% and 35%,” “between 35% and 50%,” and “more than 50%.” In addition, according to Meyer et al.’s [ 56 ] work, different types of programming tasks impact productivity differently. For instance, activities such as development and bug fixing were perceived as productive, whereas testing was considered as unproductive. We also asked participants about the percentage of time they spent on different programming themes in off-hours, using the same options as in the previous question. We asked participants to specify the detailed information if they had been involved in activities or programming theme other than those we listed.

Moreover, to understand whether developers believe extra working hours can contribute to productivity, we included a question asking whether extra working hours increase productivity. Participants were given the option to select either “agree,” “neutral,” or “disagree.” Then, we cross checked their ideas with their motivations for working overtime. Beckers et al. [ 57 ] proposed that the outcome of extra working hours was affected by motivation. Highly motivated workers might have more active attitude towards extra working hours. To see how participants’ perspectives on extra working hours differ with motivations, we included a multiple-choice question, listing nine common reasons for working overtime. These options were derived from initial interviews with several developers, who explained why they worked overtime. Their reasons were used as initial options in pilot tests. During the pilot tests, participants were asked to provide additional reasons if theirs were not listed. We then reviewed their answers and adjusted the options to ensure that the given reasons covered all cases. Finally, we concluded nine reasons from their responses, such as (1) handling emergencies (such as application crashes), (2) meeting deadlines, (3) making up for the time wasted on programming-independent work activities during office hours, (4) taxi reimbursement (some companies covered the taxi expenses within specific hours), (5) good environment of company (such as free snacks and air conditioners), (6) peer pressure (participants mentioned they stayed in the office after work because most of their colleagues did not leave), (7) company requirements, (8) enjoying coding in spare time, and (9) working for bonus. One or more options could be selected. Participants could also specify their reasons if they are not given as options.

5.1. RQ1. What Are the Work Rhythms of Projects and Developers?

5.1.1. work rhythms of developers.

We apply clustering analysis on the commit behavior of developers in our dataset. Four work rhythms are detected among the developers in our dataset. We visualize the four detected work rhythms in the form of heatmap, as shown in Figures 3(a) , 3(b) , 3(c) , and 3(d) , with the x -axis representing the hours and the y -axis representing the days in a week. The color intensity of each time slot shows the aggregated commit frequency among developers, where darker color indicates higher commit frequencies. The detected work rhythms exhibit unique characteristics. The 48 hr in weekdays and weekends are divided into four subsets, as shown in Table 2 . We observe the commit behavior in the subsets of hours and summarize the following characteristics:

• The 48 hr in weekdays and weekends are divided into four time subsets. Developers with the same rhythm have the same degree of commit frequency in each time subset. For example, as shown in Figure 3(a) , developers with rhythm #1 made commits at a high frequency during 9 a.m. to 5 p.m. on weekdays (i.e., time subset #1), whereas they have much fewer commits during the other time subsets.

#1: Nine-to-five worker. As shown in Figure 3(a) , developers with work rhythm #1 concentrate on programming during regular office hours (9 a.m. to 5 p.m.) on weekdays. They submit code changes less frequently after work hours or on weekends.

#2: Flex timers. As shown in Figure 3(b) , the code submissions of developers with rhythm #2 are uniformly distributed on almost every hour on weekdays. Developers with this rhythm are likely to submit code changes at any time of the day and do not display fixed work and rest time.

#3: Overnight developers. As shown in Figure 3(c) , developers with rhythm #3 submit their codes from 9 a.m. to 12 a.m. They also make code submissions on weekends following a similar daily working schedule as weekday, whereas the commit frequency on weekends is lower than that on weekdays.

#4: Off-hour developers. As shown in Figure 3(d) , the peak time of the code submissions of developers with rhythm #4 is weekday nights and weekends, instead of regular working hours on weekdays.

5.1.2. Work Rhythms of Projects

We also apply clustering analysis on the commit behavior of repositories. Three work rhythms are detected among the repositories in our dataset. Figures 4(a) and 4(b) present the temporal distributions of commit frequency for identified rhythms. The 48 hr in weekdays and weekends are divided into three subsets, as shown in Table 3 . We summarize the features of the three identified rhythms as follows:

• The 48 hr in weekdays and weekends are divided into three subsets.

#1: Typical office hours. Repositories with work rhythm #1 adopt typical work time, usually from 9 a.m. to 5 p.m on weekdays. Code changes are rarely submitted into those repositories on weekends.

#2: Slightly extended working hours. Repositories with rhythm #2 extend the typical work time to 6 p.m. on weekdays. Compared with developers in rhythm #1, repositories with rhythm #2 usually have more code submissions on weekends.

#3: Working over night and weekend. Repositories with rhythm #3 endure longer working hours than the other two rhythms. Developers of these repositories work equally on weekdays and weekends, starting from nine in the morning to the midnight.

The percentage of developers and repositories in each detected work rhythm is shown in Figures 5(a) and 5(b) , respectively. Among the four work rhythms detected in the developer dataset, we observe that about two-thirds of the developers follow rhythm #1 (typical working hours), which conforms to Claes et al.’s [ 1 ] finding. Among the three work rhythms detected in the repository dataset, rhythm #1 covers half of the repositories and rhythm #2 takes up 40% repositories, and the rest 10% repositories follow rhythm #3.

5.2. RQ2. Are Work Rhythms Related to Demographics and Collaboration Role?

Do work rhythms vary across different regions? We examine the work regions of the developers. The percentages of developers per rhythm in each region are shown in Figure 6 . Developers working for organizations in the United States and Europe mainly follow rhythm #1, whereas rhythms #3 and #4 are more prevalent among developers working for organizations in China or “multiple countries”. We divide developers into two groups according to their work regions: the United States and Europe as a group and China and “multiple countries” as another group. We apply chi-square test to check the frequency of the two groups in each of the four rhythms. We find a significant difference between the two groups of developers in terms of the four work rhythms ( p -value < 0.001, Cramer’s V  = 0.325).

Is there a correlation between work rhythm and developer seniority? We investigate the account age of developers in each rhythm and perform Mann–Whitney U test. Figure 7(a) shows the account ages of the developers for each work rhythm in box plots. Developers with rhythms #3 ( p -value < 0.001, Cliff’s delta d  = 0.20) and #4 ( p -value = 0.004, d  = 0.13) tend to create their GitHub accounts earlier than those with other rhythms, which indicates that developers with rhythms #3 and #4 start to be engaged in software development earlier than those with the other two rhythms. Developers with rhythm #1 created their GitHub accounts later than others ( p -value < 0.001, d  = −0.20).

Is there a correlation between work rhythm and project maturity? We investigate the repository age in each rhythm and perform Mann–Whitney U test. As shown in Figure 7(b) , repositories with the three rhythms do not show significant difference in terms of repository ages ( p -values > 0.05).

Do SHS developers have specific work rhythms? The percentages of developers in each rhythm among SHS developers and ordinary developers are shown in Figure 8 . There are more developers with rhythm #1 and fewer developers with rhythm #3 among ordinary developers than among SHS developers. We apply chi-square test and find a significant difference between SHS and non-SHS developers in terms of rhythms #1 and #3 ( p -value = 0.006, Cramer’s V  = 0.128). Compared with ordinary developers, SHS developers tend to be overnight developers rather than work in fixed office hours.

5.3. RQ3. What Are the Correlations between Different Work Rhythms and Technical Performance?

Next, we examine the effects of work rhythms on various measures of technical performance. Figures 9(a) , 9(b) , and 9(c) present the performance on the three measures for developers. We perform Mann–Whitney U test and the results are shown in Table 4 . The value in each entry of the table is the ratio between the median value of the measures within the group and outside the group. A less than 1 value indicates that the developers with the selected rhythm have smaller value in the chosen measure and a higher than 1 value means otherwise. In addition,   ∗ marks the difference is significant with p -value ≤ 0.05,   ∗∗ marks p -value ≤ 0.01 and   ∗∗∗ marks p -value ≤ 0.001. As shown in Table 4 , developers with rhythms #3 and #4 had more followers (Cliff’s delta d  = 0.30 and 0.16 respectively), received more stars from their own repositories ( d  = 0.228 and 0.158 respectively) and had higher h-indexes ( d  = 0.239 and 0.169 respectively). In contrast, developers with rhythm #1 perform the worst in all three measures: average number of stars ( d  = −0.235), number of followers ( d  = −0.282), and h-index ( d  = −0.243).

•   ∗∗∗ marks difference is significant with p -value ≤ 0.001.

We also examine the effect of repositories’ work rhythms on technical performance and apply Mann–Whitney U test. The results are shown in Figures 10(a) , 10(b) , 10(c) , and 10(d) and Table 5 . Repositories with rhythm #2 receive more stars ( d  = 0.085) and have more forks ( d  = 0.090) than those with the other two rhythms. Repositories with rhythm #3 receive more stars than others ( d  = 0.151). As for the number of open issues, there is no significant difference among the three work rhythms.

•   ∗ marks the difference is significant with p -value ≤ 0.05,   ∗∗ marks p -value ≤ 0.01, and   ∗∗∗ marks p -value ≤ 0.001.

It is interesting to find that although repositories with rhythm #1 have larger LOC changed than those with the other two rhythms, their values of the other measures of technical performance including stars ( d  = −0.133) and forks ( d  = −0.10) turn out to be lower. To discover the reason for this phenomenon, we further check the number of lines of code added and deleted per commit in each hour of a day. As shown in Figures 11(a) and 11(b) , during the typical office hours, both the lines of code added and deleted per commit submitted into repositories with rhythm #1 are larger than those with the other two rhythms. During 4.–5 p.m. the sizes of the commits are the largest among commits in all hours of the day. The commit sizes peak between 4 and 5 p.m., suggesting a hypothesis that developers working on repositories with rhythm #1 may submit larger commits just before leaving the office to finish their workday on time. However, this practice might lead to lower code quality, necessitating deletions and rewrites the next day. As a result, these repositories have more frequent code changes, but their stars and forks are fewer.

5.4. RQ4. What Are Developers’ Attitudes on Work Rhythm and Productivity?

5.4.1. required working hour vs. actual working hour.

We ask participants about their companies’ required working hour and their actual working hour on a typical work day. As shown in Figure 12 , most participants reply that their companies require an 8-hr work day schedule. However, they usually work longer hours than required.

5.4.2. Content Switch between Office Hours and Off-Hours

Figure 13 presents the distribution of activities during office hours and off-hours. Coding occupies the majority of time in both periods. The rankings for time spent on different tasks are mostly consistent, except for meetings and studying. During the office hours, meetings rank the third and the sixth respectively, whereas, during the off-hours, studying moves up to the second and meetings drop to the sixth. As shown in Figure 14 , the most common programming activity during off-hours is developing, followed by testing, bug fixing, and creating backups.

5.4.3. Perspectives on Productivity in Extra Working Hours

Except for 25 participants (27.17%) who claim they do not work extra hours, 38 participants (41.30%) believe that additional working hours enhance productivity, 26 participants (28.26%) believe that additional work time does not boost productivity, and three participants (3.26%) are neutral.

We ask participants why they work overtime. Among all the options, “deadline” receives the most votes (33.3%). “Emergency” is the second most popular reason with 32.3% responses. In addition, 24.7% mention that they work overtime to make up for the time wasted on programming-independent work activities during office hours, 19.4% say that their companies require extra working hours, 16.1% agree that they work overtime because of peer pressure, 15.1% claim that they work overtime because they enjoy coding in their spare time, 7.5% say that they stay in the office after work because their companies provide good environment, and 6.5% mention that they work overtime because their companies provide taxi reimbursement. Only 1.1% say because of the bonus that their companies offer for overtime work.

We cross-check their motivations and views on the productivity of additional working hours. The results are shown in Figure 15 , in which the height of a rectangle represents the proportion of participants who agree on the option and the flow represents the proportion of participants who agree on both the two options on each side. According to the results, more respondents agree extra working hours could increase productivity if they work overtime for emergencies (19 agree and 8 disagree), deadlines (18 agree and 10 disagree), making up for the time wasted on programming-independent work activities (13 agree and 10 disagree), taxi reimbursement (4 agree and 2 disagree), or good environment of their companies (3 agree and 2 disagree). In contrast, fewer respondents agree with the idea if they work overtime because of the company’s requirements (8 agree and 9 disagree), peer pressures (7 agree and 8 disagree), or bonus (0 agrees and 1 disagrees). Among the respondents who work overtime because they enjoy coding in their spare time, the numbers of participants who hold both views are the same (4 agree and 4 disagree).

6. Discussion

6.1. implications for software practice.

The purpose of this paper is to investigate the work rhythms in software development and their effects on technical performance. We identify four typical work rhythms in the developer dataset. The typical working hours (from 9 a.m. to 5 p.m. on weekdays) cover 64% of developers in the dataset. The rest three rhythms represent an aperiodic work rhythm, an overnight work rhythm, and an off-hour work rhythm, respectively. In addition, three work rhythms are detected among repositories in the dataset. There are one typical work rhythm covering half of the repositories and two different types of overtime work rhythm.

Work rhythms are correlated with demographics and collaboration roles. Work rhythms with moderately extended working hours are more popular among senior developers. The maturity of a repository does not decrease the chance of requiring its developers to work extra hours. Developers who bridge collaboration groups consist of a higher proportion of “overnight developers” than others.

Work rhythms with a moderate amount of extended working hours appear to be associated with good technical performance. According to our results, projects and developers following the work rhythms with moderate hours of overtime work (rhythms #3 and #4 in developers’ rhythms and rhythms #2 and #3 in repositories’ rhythms) turn out to have better work performance than those following other rhythms. Projects and developers following fixed-hour work rhythms (rhythm #1 in developers’ rhythms and rhythm #1 in repositories’ rhythms) show poorer technique performance. Developers who follow aperiodic work rhythm (rhythm #2 in developers’ rhythms) do not present better performance than others.

Developers’ perspectives on productivity in extended working hours are influenced by their motivations of working overtime. They would feel extended working hours increase their productivity when the time for coding is insufficient due to some unexpected arrangments (such as approaching deadline) or the companies give clear incentives (such as reimbursing taxi fares), while fewer believe that extended working hours could increase productivity if they are under the requirement of companies, or work for bonus, or just follow the other colleagues to work overtime. Tech companies and teams could benefit from practices, for example, not forcing the members to work extra hours, and providing employees with better work environment and clear incentives.

6.2. Limitations and Threats to Validity

Being a first study to reveal work rhythms in software development and their effects on technical performance, there are a few limitations in our work. First, the data analysis in our study is limited to public open-source projects hosted on GitHub. Therefore, our conclusions are specific to the open-source projects and their contributors. Although our findings demonstrate notable distinctions between work rhythms, we cannot guarantee their broader applicability to the entire industry, as comprehensive data on a wider range of companies and closed-source projects would be necessary. We notice that there are alternative platforms such as GitLab where organizations release their work projects in a timely way. In addition, while we aim to capture an authentic snapshot of developer activity in open-source projects by forming an actual distribution of repositories in the companies, the variation in the number of repositories across these companies could potentially introduce bias into the results. In future work, we plan to explore other data sources to validate and expand our findings.

Second, our quantitative analysis on the work rhythms primarily focuses on the commit activities. Analysis on more comprehensive dataset could better reveal of rules of one field of research [ 58 ]. Other activities, such as meetings and document writing, also occupy developers’ working hours; therefore, the time spent on programming might not fully represent their work schedule. However, because programming is a major task of developers’ work, the temporal pattern of commits is a strong indicator of work time and our findings could provide insights into developers’ working status. We also acknowledge that there might be a delay between the time of making commits and the actual time of completing coding tasks. However, because our analysis is based on aggregated commits rather than individual ones, the impact of such delays should be negligible.

Third, the metrics that we use to measure the technical performance are indirect. For developers, we use average number of stars, number of followers, and h-index of stars as indicators of their reputations. For repositories, we consider number of stars, number of forks, and number of issues as proxies for user attention. More user attention and discussions mean that the repositories and developers are recognized by more people, which indicates their good technical performance. In addition, we use the lines of code changed per week to measure code productivity. Although these measures are intuitively reasonable, they could only show technical performance in some way. More metrics such as code quality should be addressed to obtain a comprehensive understanding of the technical performance.

7. Conclusions and Future Work

In this paper, we aim to discover work rhythms in software development and investigate their effects on technical performance. We found four work rhythms among individuals and three work rhythms among repositories in our dataset. The findings indicate that developers working for organizations in China or multiple countries tend to follow long-hour work rhythms, whereas those working for organizations in the United States and Europe tend to follow the typical work rhythm. Regarding the effects of work rhythms on technical performance, we found that a moderate amount of overtime work is related to good technical performance, whereas fixed office hours appear to be associated with projects and developers who receive less attention. In addition, our survey study indicates that developers usually tend to work longer than their companies’ required working hours. A positive attitude towards overtime work is often linked to situations that require addressing unexpected issues, such as approaching deadlines, or when clear incentives are provided.

For future work, we aim to delve deeper into the underlying mechanisms behind developers’ work. We wish to understand the underlying causes for different working rhythms by considering the interplay between work rhythms and other factors, such as technical roles and collaboration patterns. Furthermore, we plan to investigate the causal relationship between work rhythms and technical performance by conducting experimentation and incremental studies.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This work has been sponsored by National Natural Science Foundation of China (nos. 62072115 and 62102094), Shanghai Science and Technology Innovation Action Plan Project (no. 22510713600), European Union’s Horizon 2020 Research and Innovation Programme under the grant agreement no. 101021808, and Marie Skłodowska Curie grant agreement no. 956090.

The User Survey

What is the country of your company?

How long have you been employed at your current company?

What is the type of your current job? (e.g., development, testing, product management, etc.)

What is your company’s designated working hour for workdays? (Please fill in the start and end time in 24-hr format.)

What are your actual working hour for workdays? (Please fill in the start and end time in 24-hr format.)

I work on both Saturday and Sunday every weekend

I work on either Saturday or Sunday every weekend

I sometimes work on weekends (less than once a week, please specify how many days per month on average)

I never work on weekends

Other (please specify)

Most of my colleagues work overtime.

My company provides benefits for overtime worker.

I enjoy working overtime.

I work during holidays.

I work more before/after holidays.

Project planning

Reading/writing documents and preparing reports

Handling other work tasks, e.g., reading/writing emails, etc.

Learning software, tools, skills, etc.

Business entertainment, e.g., hosting colleagues, etc.

Leisure activities

Development

I do not work overtime.

Handling emergencies (such as application crashes).

Making up for the time wasted on programming-independent work activities during office hours.

Company requirements.

Peer pressure (most of my colleagues have not left).

Enjoying coding in spare time.

The company provides good environment, e.g., free smacks and air conditioners.

The company provides taxi reimbursements within specific hours.

Working for bonus.

Other. (Please specify the reason.)

Agree—overall, working overtime increases my work output.

Disagree—overtime work does not compensate for my extra working hours.

Open Research

Data availability.

As the data used in this work are publicly visible and accessible on GitHub, researchers interested in accessing the data can retrieve it directly from the GitHub platform with its official API. To ensure transparency and facilitate further research, the list of organizations and repositories in our dataset is publicly available on GitHub: https://github.com/jiayunz/Work-Rhythms-in-Software-Development . Researchers can refer to this repository to gain access to the specific projects and repositories included in the dataset. For any inquiries or requests related to the dataset, researchers can contact the corresponding author through email.

• 1 Claes M. , Mäntylä M. , Kuutila M. , and Adams B. , Do programmers work at night or during the weekend? , Proceedings of the 40th International Conference on Software Engineering , 2018, IEEE, 705 – 715 . Google Scholar
• 2 Marlow J. and Dabbish L. , Activity traces and signals in software developer recruitment and hiring , Proceedings of the 2013 Conference on Computer Supported Cooperative Work , 2013, ACM, 145 – 156 . Google Scholar
• 3 Baltes B. B. , Briggs T. E. , Huff J. W. , Wright J. A. , and Neuman G. A. , Flexible and compressed workweek schedules: a meta-analysis of their effects on work-related criteria , Journal of Applied Psychology . ( 1999 ) 84 , no. 4, 496 – 513 , https://doi.org/10.1037/0021-9010.84.4.496 , 2-s2.0-0033247275. 10.1037/0021-9010.84.4.496 Web of Science® Google Scholar
• 4 Smith L. , Folkard S. , Tucker P. , and Macdonald I. , Work shift duration: a review comparing eight hour and 12 hour shift systems , Occupational and Environmental Medicine . ( 1998 ) 55 , no. 4, 217 – 229 , https://doi.org/10.1136/oem.55.4.217 , 2-s2.0-0031917695. 10.1136/oem.55.4.217 CAS PubMed Web of Science® Google Scholar
• 5 Krueger G. P. , Sustained work, fatigue, sleep loss and performance: a review of the issues , Work & Stress . ( 1989 ) 3 , no. 2, 129 – 141 , https://doi.org/10.1080/02678378908256939 , 2-s2.0-0024318997. 10.1080/02678378908256939 Web of Science® Google Scholar
• 6 Josten E. J. , Ng-A-Tham J. E. , and Thierry H. , The effects of extended workdays on fatigue, health, performance and satisfaction in nursing , Journal of Advanced Nursing . ( 2003 ) 44 , no. 6, 643 – 652 , https://doi.org/10.1046/j.0309-2402.2003.02854.x , 2-s2.0-0346728933. 10.1046/j.0309-2402.2003.02854.x PubMed Web of Science® Google Scholar
• 7 Lockley S. W. , Barger L. K. , Ayas N. T. , Rothschild J. M. , Czeisler C. A. , and Landrigan C. P. , Effects of health care provider work hours and sleep deprivation on safety and performance , The Joint Commission Journal on Quality and Patient Safety . ( 2007 ) 33 , no. 11, 7 – 18 , https://doi.org/10.1016/S1553-7250(07)33109-7 , 2-s2.0-36649031201. 10.1016/S1553-7250(07)33109-7 PubMed Google Scholar
• 8 Richardson A. , Turnock C. , Harris L. , Finley A. , and Carson S. , A study examining the impact of 12-hour shifts on critical care staff , Journal of Nursing Management . ( 2007 ) 15 , no. 8, 838 – 846 , https://doi.org/10.1111/j.1365-2934.2007.00767.x , 2-s2.0-35348959451. 10.1111/j.1365-2934.2007.00767.x CAS PubMed Google Scholar
• 9 Keller S. M. , Berryman P. , and Lukes E. , Effects of extended work shifts and shift work on patient safety, productivity, and employee health , AAOHN Journal . ( 2009 ) 57 , no. 12, 497 – 504 , https://doi.org/10.1177/216507990905701204 . 10.1177/216507990905701204 PubMed Web of Science® Google Scholar
• 10 Eyolfson J. , Tan L. , and Lam P. , Do time of day and developer experience affect commit bugginess? , Proceedings of the 8th Working Conference on Mining Software Repositories , 2011, ACM, 153 – 162 . Google Scholar
• 11 Eyolfson J. , Tan L. , and Lam P. , Correlations between bugginess and time-based commit characteristics , Empirical Software Engineering . ( 2014 ) 19 , no. 4, 1009 – 1039 , https://doi.org/10.1007/s10664-013-9245-0 , 2-s2.0-84901846775. 10.1007/s10664-013-9245-0 Google Scholar
• 12 Prechelt L. and Pepper A. , Why software repositories are not used for defect-insertion circumstance analysis more often: a case study , Information and Software Technology . ( 2014 ) 56 , no. 10, 1377 – 1389 , https://doi.org/10.1016/j.infsof.2014.05.001 , 2-s2.0-84905091993. 10.1016/j.infsof.2014.05.001 Web of Science® Google Scholar
• 13 Kluger Y. , Basri R. , Chang J. T. , and Gerstein M. , Spectral biclustering of microarray data: coclustering genes and conditions , Genome Research . ( 2003 ) 13 , no. 4, 703 – 716 , https://doi.org/10.1101/gr.648603 , 2-s2.0-0037399130. 10.1101/gr.648603 CAS PubMed Web of Science® Google Scholar
• 14 Burt R. S. , Structural Holes: The Social Structure of Competition , 2009 , Harvard University Press. Google Scholar
• 15 Perry D. E. , Staudenmayer N. A. , and Votta L. G. , Understanding and improving time usage in software development , Software Process . ( 1995 ) 5 , 111 – 135 . Google Scholar
• 16 LaToza T. D. , Venolia G. , and DeLine R. , Maintaining mental models: a study of developer work habits , Proceedings of the 28th International Conference on Software Engineering , 2006, ACM, 492 – 501 . Google Scholar
• 17 Fu E. , Zhuang Y. , Zhang J. , Zhang J. , and Chen Y. , Understanding the user interactions on GitHub: a social network perspective , Proceedings of CSCWD , 2021, IEEE, 1148 – 1153 . Google Scholar
• 18 Gong Q. , Chen Y. , He X. , Zhuang Z. , Wang T. , Huang H. , Wang X. , and Fu X. , DeepScan: exploiting deep learning for malicious account detection in location-based social networks , IEEE Communications Magazine . ( 2018 ) 56 , no. 11, 21 – 27 , https://doi.org/10.1109/MCOM.2018.1700575 , 2-s2.0-85054599203. 10.1109/MCOM.2018.1700575 Web of Science® Google Scholar
• 19 He X. , Gong Q. , Chen Y. , Zhang Y. , Wang X. , and Fu X. , DatingSec: detecting malicious accounts in dating apps using a content-based attention network , IEEE Transactions on Dependable and Secure Computing . ( 2021 ) 18 , no. 5, 2193 – 2208 , https://doi.org/10.1109/TDSC.8858 . 10.1109/TDSC.8858 Web of Science® Google Scholar
• 20 Saini M. and Kaur K. , Fuzzy analysis and prediction of commit activity in open source software projects , IET Software . ( 2016 ) 10 , no. 5, 136 – 146 , https://doi.org/10.1049/iet-sen.2015.0087 , 2-s2.0-84990211234. 10.1049/iet-sen.2015.0087 Web of Science® Google Scholar
• 21 Javeed F. , Siddique A. , Munir A. , Shehzad B. , and Lali M. I. U. , Discovering software developer’s coding expertise through deep learning , IET Software . ( 2020 ) 14 , no. 3, 213 – 220 , https://doi.org/10.1049/iet-sen.2019.0290 . 10.1049/iet-sen.2019.0290 Web of Science® Google Scholar
• 22 Aljemabi M. A. , Wang Z. , and Saleh M. A. , Mining social collaboration patterns in developer social networks , IET Software . ( 2020 ) 14 , no. 7, 839 – 849 , https://doi.org/10.1049/iet-sen.2019.0316 . 10.1049/iet-sen.2019.0316 Google Scholar
• 23 Sajedi-Badashian A. and Stroulia E. , Investigating the information value of different sources of evidence of developers’ expertise for bug assignment in open-source projects , IET Software . ( 2020 ) 14 , no. 7, 748 – 758 , https://doi.org/10.1049/iet-sen.2019.0384 . 10.1049/iet-sen.2019.0384 Google Scholar
• 24 Traullé B. and Dalle J.-M. , The evolution of developer work rhythms , International Conference on Social Informatics , 2018, Springer, 420 – 438 . Google Scholar
• 25 Zhang J. , Chen Y. , Gong Q. , Wang X. , Ding A. Y. , Xiao Y. , and Hui P. , Understanding the working time of developers in IT companies in China and the United States , IEEE Software . ( 2021 ) 38 , no. 2, 96 – 106 , https://doi.org/10.1109/MS.2020.2988022 . 10.1109/MS.2020.2988022 Google Scholar
• 26 Khomh F. , Adams B. , Dhaliwal T. , and Zou Y. , Understanding the impact of rapid releases on software quality , Empirical Software Engineering . ( 2015 ) 20 , no. 2, 336 – 373 , https://doi.org/10.1007/s10664-014-9308-x , 2-s2.0-84928707650. 10.1007/s10664-014-9308-x Google Scholar
• 27 Gong Q. , Zhang J. , Chen Y. , Li Q. , Xiao Y. , Wang X. , and Hui P. , Detecting malicious accounts in online developer communities using deep learning , Proceedings of the 28th ACM International Conference on Information and Knowledge Management , 2019, 1251 – 1260 . Google Scholar
• 28 Gong Q. , Liu Y. , Zhang J. , Chen Y. , Li Q. , Xiao Y. , Wang X. , and Hui P. , Detecting malicious accounts in online developer communities using deep learning , IEEE Transactions on Knowledge and Data Engineering . ( 2023 ) 35 , no. 10, 10633 – 10649 , https://doi.org/10.1109/TKDE.2023.3237838 . 10.1109/TKDE.2023.3237838 Google Scholar
• 29 Goyal R. , Ferreira G. , Kästner C. , and Herbsleb J. , Identifying unusual commits on GitHub , Journal of Software: Evolution and Process . ( 2018 ) 30 , no. 1, e1893. 10.1002/smr.1893 Web of Science® Google Scholar
• 30 MacQueen J. , Classification and analysis of multivariate observations , Fifth Berkeley Symposium on Mathematical Statistics and Probability , 1967, Los Angeles, LA, USA, University of California, 281 – 297 . Google Scholar
• 31 Ester M. , Kriegel H.-P. , Sander J. , and Xu X. , A density-based algorithm for discovering clusters in large spatial databases with noise , 2nd International Conference on Knowledge Discovery and Data Mining , 1996, ACM, 226 – 231 . Google Scholar
• 32 Cheng Z. , Trépanier M. , and Sun L. , Probabilistic model for destination inference and travel pattern mining from smart card data , Transportation . ( 2021 ) 48 , no. 4, 2035 – 2053 , https://doi.org/10.1007/s11116-020-10120-0 . 10.1007/s11116-020-10120-0 Web of Science® Google Scholar
• 33 Li Z. , Yan H. , Zhang C. , and Tsung F. , Individualized passenger travel pattern multi-clustering based on graph regularized tensor latent dirichlet allocation , Data Mining and Knowledge Discovery . ( 2022 ) 36 , no. 4, 1247 – 1278 , https://doi.org/10.1007/s10618-022-00842-3 . 10.1007/s10618-022-00842-3 Google Scholar
• 34 Pedregosa F. , Varoquaux G. , Gramfort A. , Michel V. , Thirion B. , Grisel O. , Blondel M. , Prettenhofer P. , Weiss R. , Dubourg V. , Vanderplas J. , Passos A. , Cournapeau D. , Brucher M. , Perrot M. , and Duchesnay É. , Scikit-learn: machine learning in python , Journal of Machine Learning Research . ( 2011 ) 12 , no. Oct, 2825 – 2830 . Web of Science® Google Scholar
• 35 Vasilescu B. , Yu Y. , Wang H. , Devanbu P. , and Filkov V. , Quality and productivity outcomes relating to continuous integration in GitHub , Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering , 2015, ACM, 805 – 816 . Google Scholar
• 36 Burt R. S. , Kilduff M. , and Tasselli S. , Social network analysis: foundations and frontiers on advantage , Annual Review of Psychology . ( 2013 ) 64 , no. 1, 527 – 547 , https://doi.org/10.1146/annurev-psych-113011-143828 , 2-s2.0-84872461605. 10.1146/annurev-psych-113011-143828 PubMed Web of Science® Google Scholar
• 37 Lin Z. , Zhang Y. , Gong Q. , Chen Y. , Oksanen A. , and Ding A. Y. , Structural hole theory in social network analysis: a review , IEEE Transactions on Computational Social Systems . ( 2022 ) 9 , no. 3, 724 – 739 , https://doi.org/10.1109/TCSS.2021.3070321 . 10.1109/TCSS.2021.3070321 Web of Science® Google Scholar
• 38 Li W. , Xu Z. , Sun Y. , Gong Q. , Chen Y. , Ding A. Y. , Wang X. , and Hui P. , DeepPick: a deep learning approach to unveil outstanding users with public attainable features , IEEE Transactions on Knowledge and Data Engineering . ( 2023 ) 35 , no. 1, 291 – 306 . Google Scholar
• 39 Bhowmik T. , Niu N. , Singhania P. , and Wang W. , On the role of structural holes in requirements identification: an exploratory study on open-source software development , ACM Transactions on Management Information Systems . ( 2015 ) 6 , no. 3, 1 – 30 , https://doi.org/10.1145/2795235 , 2-s2.0-84942051662. 10.1145/2795235 Google Scholar
• 40 Xu W. , Rezvani M. , Liang W. , Yu J. X. , and Liu C. , Efficient algorithms for the identification of top- structural hole spanners in large social networks , IEEE Transactions on Knowledge and Data Engineering . ( 2017 ) 29 , no. 5, 1017 – 1030 . 10.1109/TKDE.2017.2651825 Web of Science® Google Scholar
• 41 Gong Q. , Zhang J. , Wang X. , and Chen Y. , Identifying structural hole spanners in online social networks using machine learning , Proceedings of the ACM SIGCOMM. 2019 Conference Posters and Demos , 2019, 93 – 95 . Google Scholar
• 42 Gao M. , Li Z. , Li R. , Cui C. , Chen X. , Ye B. , Li Y. , Gu W. , Gong Q. , Wang X. , and Chen Y. , EasyGraph: a multifunctional, cross-platform, and effective library for interdisciplinary network analysis , Patterns . ( 2023 ) 4 , no. 10, https://doi.org/10.1016/j.patter.2023.100839 , 100839. 10.1016/j.patter.2023.100839 Google Scholar
• 43 Tsay J. , Dabbish L. , and Herbsleb J. , Influence of social and technical factors for evaluating contribution in GitHub , Proceedings of the 36th International Conference on Software Engineering , 2014, ACM, 356 – 366 . Google Scholar
• 44 Dabbish L. , Stuart C. , Tsay J. , and Herbsleb J. , Social coding in GitHub: transparency and collaboration in an open software repository , Proceedings of the ACM. 2012 conference on Computer Supported Cooperative Work , 2012, ACM, 1277 – 1286 . Google Scholar
• 45 Hirsch J. E. , An index to quantify an individual’s scientific research output , Proceedings of the National Academy of Sciences of the United States of America . ( 2005 ) 102 , no. 46, 16569 – 16572 , https://doi.org/10.1073/pnas.0507655102 , 2-s2.0-28044445101. 10.1073/pnas.0507655102 CAS PubMed Web of Science® Google Scholar
• 46 Gong Q. , Chen Y. , He X. , Xiao Y. , Hui P. , Wang X. , and Fu X. , Cross-site prediction on social influence for cold-start users in online social networks , ACM Transactions on the Web . ( 2021 ) 15 , no. 2, 1 – 23 , https://doi.org/10.1145/3409108 . 10.1145/3409108 Google Scholar
• 47 Borges H. , Hora A. , and Valente M. T. , Understanding the factors that impact the popularity of GitHub repositories , 2016 IEEE International Conference on Software Maintenance and Evolution , 2016, IEEE, 334 – 344 . Google Scholar
• 48 Jiang J. , Lo D. , He J. , Xia X. , Kochhar P. S. , and Zhang L. , Why and how developers fork what from whom in GitHub , Empirical Software Engineering . ( 2017 ) 22 , no. 1, 547 – 578 , https://doi.org/10.1007/s10664-016-9436-6 , 2-s2.0-84973174936. 10.1007/s10664-016-9436-6 Web of Science® Google Scholar
• 49 Tsay J. , Dabbish L. , and Herbsleb J. , Let’s talk about it: evaluating contributions through discussion in GitHub , Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , 2014, ACM. Google Scholar
• 50 Vale G. , Schmid A. , Santos A. R. , De Almeida E. S. , and Apel S. , On the relation between GitHub communication activity and merge conflicts , Empirical Software Engineering . ( 2020 ) 25 , no. 1, 402 – 433 , https://doi.org/10.1007/s10664-019-09774-x , 2-s2.0-85073927730. 10.1007/s10664-019-09774-x Google Scholar
• 51 Vasilescu B. , Blincoe K. , Xuan Q. , Casalnuovo C. , Damian D. , Devanbu P. , and Filkov V. , The sky is not the limit: multitasking across GitHub projects , Proceedings of the 38th International Conference on Software Engineering , 2016, IEEE, 994 – 1005 . Google Scholar
• 52 Dieste O. , Aranda A. M. , Uyaguari F. , Turhan B. , Tosun A. , Fucci D. , Oivo M. , and Juristo N. , Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study , Empirical Software Engineering . ( 2017 ) 22 , no. 5, 2457 – 2542 , https://doi.org/10.1007/s10664-016-9471-3 , 2-s2.0-85011654161. 10.1007/s10664-016-9471-3 Web of Science® Google Scholar
• 53 Oliveira E. , Fernandes E. , Steinmacher I. , Cristo M. , Conte T. , and Garcia A. , Code and commit metrics of developer productivity: a study on team leaders perceptions , Empirical Software Engineering . ( 2020 ) 25 , no. 4, 2519 – 2549 , https://doi.org/10.1007/s10664-020-09820-z . 10.1007/s10664-020-09820-z Web of Science® Google Scholar
• 54 Pearson K. , X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling , The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science . ( 2009 ) 50 , no. 302, 157 – 175 , https://doi.org/10.1080/14786440009463897 . 10.1080/14786440009463897 Google Scholar
• 55 Mann H. B. and Whitney D. R. , On a test of whether one of two random variables is stochastically larger than the other , The Annals of Mathematical Statistics . ( 1947 ) 18 , no. 1, 50 – 60 . 10.1214/aoms/1177730491 Web of Science® Google Scholar
• 56 Meyer A. N. , Fritz T. , Murphy G. C. , and Zimmermann T. , Software developers’ perceptions of productivity , Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , 2014, ACM, 19 – 29 . Google Scholar
• 57 Beckers D. G. J. , van der Linden D. , Smulders P. G. W. , Kompier M. A. J. , van Veldhoven M. J. P. M. , and van Yperen N. W. , Working overtime hours: relations with fatigue, work motivation, and the quality of work , Journal of Occupational and Environmental Medicine . ( 2004 ) 46 , no. 12, 1282 – 1289 . PubMed Web of Science® Google Scholar
• 58 Wu J. , Ye B. , Gong Q. , Oksanen A. , Li C. , Qu J. , Tian F. F. , Li X. , and Chen Y. , Characterizing and understanding development of social computing through DBLP: a data-driven analysis , Journal of Social Computing . ( 2022 ) 3 , no. 4, 287 – 302 , https://doi.org/10.23919/JSC.2022.0018 . 10.23919/JSC.2022.0018 Google Scholar

Information

Log in to Wiley Online Library

Change password, your password must have 10 characters or more:.

• a lower case character, 
• an upper case character, 
• a special character 

Password Changed Successfully

Your password has been changed

Create a new account

Forgot your password.

Enter your email address below.

Please check your email for instructions on resetting your password. If you do not receive an email within 10 minutes, your email address may not be registered, and you may need to create a new Wiley Online Library account.

Request Username

Can't sign in? Forgot your username?

Enter your email address below and we will send you your username

If the address matches an existing account you will receive an email with instructions to retrieve your username

• Share on twitter
• Share on facebook

Global Academic Reputation Survey 2024 launching soon

Survey results will help to fuel  the ’s series of rankings in 2024.

• Share on linkedin
• Share on mail

The world’s largest invitation-only academic opinion survey, Times Higher Education ’s global Academic Reputation Survey, is being launched in early November.

The Academic Reputation Survey 2024, available in 12 languages, will be distributed by THE and use benchmark data as a guide to ensure that the response coverage is as representative of world scholarship as possible. 

The annual questionnaire targets only experienced, published scholars, who offer their views on excellence in research and teaching within their disciplines and at institutions with which they are familiar. Invitations will be spread across a three-month period. 

The survey results will help to fuel THE ’s series of rankings in 2024. 

Scholars are asked to use their discipline-specific knowledge to name up to 15 universities that they believe are the best in both research and teaching. The survey, which typically takes up to 15 minutes to complete, will close at the end of January 2024. 

The headline results of the survey will be shared with respondents. More detailed analysis will then be published in THE ’s World Reputation Rankings 2024. The survey data from 2023 and 2024 will also be used alongside 15 objective indicators to help create the THE World University Rankings 2025, to be published in late 2024, and all WUR subsidiary rankings.

The survey also provides a uniquely rich picture of the changing global academic reputation of institutions to inform THE ’s editorial analyses and data and analytics tools. 

The survey is strictly invitation-only; universities cannot make nominations or supply contact lists, and individuals cannot nominate themselves for participation. 

Please check your inbox for an invitation from [email protected] .

If you are selected to take part in the survey, you have been chosen based on a proven record of research publication and will be representing thousands of your peers in your discipline and your country. Please take the opportunity to provide your expert input and help us develop a uniquely rich perspective on global higher education.

Read the World Reputation Rankings 2023 methodology

Register to continue

Why register?

• Registration is free and only takes a moment
• Once registered, you can read 3 articles a month
• Sign up for our newsletter

Or subscribe for unlimited access to:

• Unlimited access to news, views, insights & reviews
• Digital editions
• Digital access to THE’s university and college rankings analysis

Already registered or a current subscriber? Login

Related articles

The craft of reputation building and maintenance

Institutions must be more strategic about positioning themselves and their work when considering new markets and partnerships, says Tania Rhodes-Taylor

You might also like

Impact Rankings 2025: time to register

Universities can now sign up to participate ahead of data collection opening in the autumn

Record number of universities submit data for World University Rankings

African participation surpasses North America for the first time

Impact Rankings 2024: universities make trustworthy stewards

Locale determines the mission of many universities, and the Impact Rankings recognise how such diversity and community-mindedness helps to further progress towards the SDGs

Impact Rankings 2024: who is excelling at stewardship and outreach?

New thematic analysis reveals the countries and institutions that are walking the talk on sustainability 

Featured jobs

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved August 29, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”